an HTTP server that downloads blocklists in p2p format and merges them on demand
An HTTP server written with NodeJS that downloads blocklists in p2p format and merges them on demand.
Many network-related programs support blocking ranges of IP addresses from a text file that is periodically downloaded from a web server. Unfortunately, there are scenarios in which the desired IP ranges come from separated sources on the Internet, and some programs only allow for one source url to update their IP ranges.
njs-blocklist-merger is an HTTP server that can download multiple IP range files in p2p format (see http://sourceforge.net/p/peerguardian/wiki/dev-blocklist-format-p2p/) from the web simultaneously and merge them on demand. It is meant to be run locally (eg. on http://localhost:1337/) so that programs requiring a unique IP blocklist can query it to download IP ranges from many sources.
Sets of blocklist sources are arranged in text files and placed in a local folder. Then, njs-blocklist-merger will serve a merged version of all the blocklists in a given file when requested.
njs-blocklist-merger is a NodeJS script that setups an HTTP server listening on a configured port number and waiting for requests in the form http://host:port/listname. Then it looks for a file named listname in local folder _./lists/_.
Each file in this folder must be encoded in UTF-8 and contains a list of urls specifying where to download the blocklists to merge. Empty lines and lines starting with # are ignored by njs-blocklist-merger.
After such file is parsed, njs-blocklist-merger downloads specified blocklists in parallel. Each downloaded blocklist must be in p2p format, encoded in UTF-8 and served either in plain text or gzip format. When all requested data is available, njs-blocklist-merger merges them and responds to the HTTP request with a blocklist in p2p format encoded in UTF-8 and served in plain text with MIME type text/plain.
You must have NodeJS and NPM on your system (see http://nodejs.org/ and https://npmjs.org/). Then execute the following command:
npm install njs-blocklist-merger -g
Place list files inside the lists folder in the format specified above. Then run the server by executing the command:
blocklist-merge-server --host localhost --port 1337
You can customize the host and port numbers. Defaults are localhost and 1337.
When the server is up, any program can then request http://localhost:1337/mylist to download a merged version of the blocklists specified in file ./lists/mylist.
njs-blocklist-merger is written in TypeScript (see http://www.typescriptlang.org/). You need it installed to be able to compile ./src/server.ts:
npm install typescript -g
npm install tsd -g
cd ./src/tsd install node async
npm run-script build
or this command:
tsc --target ES5 --out ./lib/blocklist-merge-server.js ./src/blocklist-merge-server.ts