Scrapper Tools [In Development]
Its in development but I use it in all my web automation project.
After 2.0.0 it will follow semantic versioning
Handbook
Mission
- To run bot in parallel
- Minimal Setup
- To Prevent from herustics against bot detection
- Make easy to scrape
- Easy to bypass captcha (Using 2 captcha plugin)
- Using typescript to serve documentation
- VPS friendly so u can scrape over vps like digital ocean or aws without running ur computer
Examples
There are various opensource and closed source example where this plugin has been used.
- AliExpress
- 1668
- Khan Academy for offline videos
Sample Example
mkdir automate-khanacademyyarn init -yyarn add scrapper-tools @types/node ts-node typescript
Your package.json should look like this
"name": "automate-khanacademy" "version": "1.0.0" "main": "index.js" "license": "MIT" "scripts": "dev": "ts-node src/bin.ts" "start": "node dist/bin.js" "build": "tsc" "dependencies": "@types/node": "^12.7.5" "scrapper-tools": "^1.0.102" "ts-node": "^8.3.0" "typescript": "^3.6.3"
Please modify the scripts so that you can use commands like yarn dev
, yarn start
and yarn build
This project encourage you to use typescript however Javascript should work fine too.
So create tsconfig.json file with following content.
{
"compilerOptions": {
"target": "esnext",
"module": "commonjs",
"outDir": "dist",
"rootDir": "src",
"sourceMap": true,
"jsx": "react",
"esModuleInterop": true,
"resolveJsonModule": true,
"allowSyntheticDefaultImports": true,
"lib": ["dom", "esnext", "es2015", "es2016", "es2017", "es2018"]
},
"exclude": ["node_modules"],
"include": ["src/bin.ts"]
}
I have created folder src and created src/bin.ts file which will contain automation/scraping logic.
Here is sample src/bin.ts
file.
{ // You can change many other settings like using 2tickets api key to bypass captcha, width ,height etc. All of the following config are optional so don't worry if you don't use them await await await await let page = await await page await page // At the end don't forget to close browser await }
I recommend you to use prettier and eslint to make code super clear and bug free.
If you want this project to move way to forward please consider donations which can be found in the end of the page.
Intresting Findings.
Debugging Puppetter using monitor
In linux machine with cli you cannot see browser? Yeah I also used to think same but no more. You can use virtual monitor and use remote viewer to see browser. Isn't it cool?
Install xvfb
sudo apt-get install xvfb
Test running google chrome
DISPLAY=:99 google-chrome --no-sandbox
Xvfb :99 &DISPLAY=:99 xvfb-run --server-args='-screen 0 1024x768x24' yarn start
Using VNC Server
Install tightvnc
sudo apt install tightvncserver
Start the vnc server port screen 99 lets say /usr/bin/tightvncserver :99
On LocalPC use SSH Tunnel
ssh -L 5999:127.0.0.1:5999 -N -f -l username remote_ipbrew install tiger-vncvncviewer 127.0.0.1:5999
Now run script using that screen
DISPLAY=:99 yarn start
On Remote PC
Injecting Functions
I have added inject function like waiting for dom element etc which is not available in evaluation script (page.evaluate function).
await page
Licence
MIT Licence
Support
Please support me by sending payment to https://www.paypal.me/KGajurel. Please inform me if you have helped me.
Thanks
Shirshak Bajgain Google Team, Sindre Sir and dependencies author for their open source.