Node.js multithreded executer
Paralio class is the heart of your application.
/* Called after initializing all of the workers */app.on'start',/* Called after killing all of the workers */app.on'end',/* Called when consuming an item */app.on'consume',
The class for worker files.
On using the REPL
Paralio comes a built-in repl to make your (my) life easier. It lets you access the app instance and run some useful commands.
You can access the main app instance with the
self variable. I.e.:
Let's you save the output to a specified path (defaults to the cwd).
In case you want to rerun your application, here you go!
I was basically scraping some information of the wikipedia. It was taking a few minutes to run over just 54 pages. So I thought to myself "What can I do to speed it up?" and the obvious answer came to my mind: multi-threading. The idea itself was brilliant, but I ran into some problems, namely: overloading my cpu with 54 processes... Then I decided to limit the number of processes data at one time and it worked. After that some improvements came to my mind and bada bim, bada boom - Paralio was born.