node-proxybroker
Local installation
The proxybroker is implemented as typexs module. It's a framework for dynamic module invocation and integration. To use this implementation create a new directory and install the npm packages.
## Create the project directory and change to it $ mkdir proxy-project && cd proxy-project ## Initialize npm project $ npm init -y ## Mark project as typexs project, ## you need to set this key to enable the typexs functionality $ node -p "JSON.stringify({...require('./package.json'), typexs: {name:'proxy-project'}}, null, 2)" > package.mod.json && \ mv package.mod.json package.json ## Install proxybroker $ npm install proxybroker ## Create the configuration directory and file $ mkdir config$ touch config/typexs.yml
Add following content to the config/typexs.yml file:
# Configuration proxy-broker: startup: true validator: parallel: 50 judge: selftest: true remote_lookup: true remote_ip: 127.0.0.1 ip: 0.0.0.0 request: timeout: 5000 provider: startup: true parallel: 5 server: proxyserver: type: proxyserver port: 3128 enable: true timeout: 30000 # limit of search repeats if proxy failed repeatLimit: 10 broker: enable: true timeouts: incoming: 30000 forward: 2000 toProxy: true # default rotator will be used schedules: - name: fetch_proxies offset: 4h # startup: true task: name: - proxy_fetch validate: true provider: __all__ - name: revalidate offset: 30m event: name: validator_run_event workers: access: - name: TaskQueueWorker access: allow
Configuration
The configuration file must be under config/typexs.yml in the project directory.
TODO
- proxy-broker
- startup
Usage
Proxyserver
Start the server with command:
typexs server
This command starts all instances of servers defined under the root key server in the configuration file.
Fetch proxies
Command:
typexs proxy-fetch [provider] [variant] -f json|csv -validate
- provider - the name of the defined proxy provider
- variant - is the possible different proxy
- -f - output format is default 'json', the other possible value is 'csv'
- --validate - run validation of grabbed data
- --store - store data also in backend
# Shows proxy variants > typexs proxy-fetch # Scan all proxy variants and return as json (can be piped to json file ) > typexs proxy-fetch __all__ > typexs proxy-fetch __all__ --validate --store > typexs proxy-fetch __all__ > proxies.json
Validate only
Command:
typexs proxy-validate [string or filename] [-f json|csv] [--store]
Examples:
# Shows proxy variants > typexs proxy-validate '127.0.0.1:3128' > typexs proxy-validate '127.0.0.1:3128'
Server-mode
TODO
Docker
TODO
docker run -p 8080:8080 -p 8181:8181 -it proxybroker-pg
Testing
curl over proxy
$ curl -x http://localhost:3128 -L http://httpbin.org/headers
Notes
- If you running proxybroker locally or behind a NAT enviroment, you must expose the ports of the judge instance.