Twitter Scraper (Web API)
Useful tool to scrape information from twitter users and tweets.
*In order to use this scraper you need an active twitter account. Scraper will sign in and use session to scrape data
*A {username}.json file with AUTH cookies will be saved in your system TMP folder
Features
- You can use it as module or in terminal
- Scraper followers, tweets, favorites from a twitter username(public profile only)
- Scrape retweeted by, liked by, comments from a tweet id
- Results can be saved in to a CSV and JSON files
Followers List Tweets List Comments List Liked By List and more
Limits:
- retweeted by - 97 inputes (web api limit).
- liked by - 100 inputes (web api limit).
- followers, comments, tweets, favorites - unlimited
Possible errors
- you can receive "rate limit" error, in that case wait and increase timeout
Installation
Install from NPM
$ npm i -g twitter-buddy
Install from YARN
$ yarn global add twitter-buddy
USAGE
In Terminal
$ twitter-buddy --help Usage: twitter-buddy <command> [options] Commands: twitter-buddy retweetedBy [id] users who retweeted specific tweet twitter-buddy likedBy [id] users who liked specific tweet twitter-buddy conversation [id] tweet comments twitter-buddy followers [id] user followers twitter-buddy tweets [id] user tweets twitter-buddy favorites [id] user favorites/likes Options: --help, -h help [boolean] --version Show version number [boolean] --login-username, -u Your twitter username [string] [default: ""] --login-password, -p Your twitter password [string] [default: ""] --number, -n Number of inputs to scrape [default: 20] --csv Save to a CSV file? [boolean] [default: true] --json Save to a JSON file? [boolean] [default: false] --timeout If the "rate limit" error received or emtpy result then you can try to set {timeout}. Timeout is in milliseconds: 1000 = 1 second [default: 0]Examples: twitter-buddy followers TWITTER_USERNAME -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 100 twitter-buddy retweetedBy TWEET_ID -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 100
Example 1 Scrape 2000 followers from twitter profile 'realDonaldTrump' and save everything to a CSV file
$ twitter-buddy followers realDonaldTrump -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 2000 CSV file saved: /{FOLDER_PATH}/followers_1571390379072_realDonaldTrump.csv
Example 2 Scrape 1000 tweets from twitter profile 'realDonaldTrump'.
$ twitter-buddy tweets realDonaldTrump -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 1000 CSV file saved: /{FOLDER_PATH}/tweets_1571390379072_realDonaldTrump.csv
Example 3 Scrape 100 users who retweeted this tweet 'https://twitter.com/realDonaldTrump/status/1183796209942695938'.
$ twitter-buddy retweetedBy 1183796209942695938 -u TWITTER_USERNAME -p TWITTER_PASSWORD -n 100 CSV file saved: /{FOLDER_PATH}/retweetedBy_1571390379072_1183796209942695938.csv
Module
Options
{
// Twitter username
username: {string},
// Twitter password
password: {string},
//If {true} then event emitter will be enabled, you won't be able to use promise
event: {boolean},
//Timeout between requests, is usefull if the 'rate limit' error was received
timeout: {int}
//Https Proxy, for example 127.0.0.1:8000
proxy: {string}
}
Promise
const twitterScraper = require('twitter-buddy');
(async() => {
try{
let bot = new twitterScraper({
username:'YOUR_TWITTER_USERNAME',
password:'YOUR_TWITTER_PASSWORD',
//event: {boolean} Enable event emitter
//timeout: {int} Timeout between requests
//proxy: {string} Https proxy, ex: 127.0.0.1:8080
});
// Scrape 1000 followers from twitter profile 'realDonaldTrump'
let followers = await bot._input({
type: 'scrape',
action: 'followers',
input: 'realDonaldTrump',
number: 1000,
});
// Scrape 500 tweets from twitter profile 'realDonaldTrump'
let tweets = await bot._input({
type: 'scrape',
action: 'tweets',
input: 'realDonaldTrump',
number: 500,
});
// Scrape 500 favorites from twitter profile 'realDonaldTrump'
let favorites = await bot._input({
type: 'scrape',
action: 'favorites',
input: 'realDonaldTrump',
number: 500,
});
// Scrape 300 comments from tweet https://twitter.com/realDonaldTrump/status/1183796209942695938'
let favorites = await bot._input({
type: 'scrape',
action: 'conversation',
input: '1183796209942695938',
number: 300,
});
// Scrape 100 users who retweeted tweet https://twitter.com/realDonaldTrump/status/1183796209942695938'
let favorites = await bot._input({
type: 'scrape',
action: 'retweetedBy',
input: '1183796209942695938',
number: 300,
});
// Scrape 100 users who liked tweet https://twitter.com/realDonaldTrump/status/1183796209942695938'
let favorites = await bot._input({
type: 'scrape',
action: 'likedBy',
input: '1183796209942695938',
number: 300,
});
}catch(error){
console.log(error);
}
})()
Event
const twitterScraper = require('twitter-buddy');
let bot = new twitterScraper({
username:'YOUR_TWITTER_USERNAME',
password:'YOUR_TWITTER_PASSWORD',
event: true, // Enable event emitter
//timeout: {int} Timeout between requests
//proxy: {string} Https proxy, ex: 127.0.0.1:8080
});
bot._input({
type: 'scrape',
action: 'followers',
input: 'realDonaldTrump',
number: 1000,
});
bot.on('data', data => {
// data
})
bot.on('error', error => {
// error
})
License
MIT
Free Software