Google HTTP Scraper
This module lets you scrape google using HTTP requests. This don't needs Google's api to work.
Features
- Proxy support: http and socks5 with and without authentication.
- Language support: Choose which language to get SERPs from.
- Useragent support: Can set your own user agent.
- Number of results from page support.
- Can have multiple instances, each with their own cookies/proxies on the same script.
Installing
npm install google-http-scraper
Usage
The module exports the GoogleSession prototype. Use it to create multiple instances each with their own cookies/proxies/languages etc.
The constructor accepts a settings object as parameter:
settings = { saveSearches: true, //If true it will save all your searches in an array for saving later. proxy: proxyObject, //An object defining the proxy to use, if not set will not use any proxy. useragent: 'string', //An string defining the useragent to use with the requests, if not set will use the defaults. maxResults: 100, //The max search results you want, searches will use this if they don't have their own settings. language: 'lang_en', //The language to use with the searches, searches will use this if they don't have their own language set. }
proxyObject = { type: 'socks or http', //The type of proxy. host: 'hostname', //The proxy host/ip address. port: 123, //The proxy's port. username: 'username', //The proxy's authentication username if needed. password: 'password', //The proxy's authentication password if needed. }
Default user agent: 'Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0'
A GoogleSession has an async method called search, it queries google using the given parameters.
searchParams = { query: 'your search query', //A string containing your query maxResults: 10, //The number of search results you want language: 'lang_en', //The language to perform the search in }
Example
const GoogleSession = require('google-http-scraper'); async function main() { var session = new GoogleSession({language: 'lang_en', maxResults: 10}); var results1 = await session.search({query: 'searching something'}); var results2 = await session.search({query: 'searching in another language', language: 'lang_pt'}); var results3 = await session.search({query: 'getting more results', maxResults: 100}); session.save('./session.json'); } main();
Author
Ícaro Augusto
- https://icaroaugusto.com
- Discord: IcaroAugusto#3303
Changelogs
1.0.0
Created.