cluster-map
Abstracts execution of tasks in parallel using Node.js cluster.
It is a high level abstraction around a common pattern used to delegate a list of tasks to the workers.
API
; /** * Executes tasks in parallel using Node.js cluster. * @typedef * @param * @returns */ /** * @typedef * @property * @property * @property */ /** * Used to create a pre-configured instance of `clusterMap`. * @param * @param * @returns */;
handleTask
function is used to receive tasks and respond to the master.
; /** * Handles a task and returns a promise that is resolved with the result of the task. * @typedef * @param * @returns */ /** * @param */;
Communication With Worker
; ;
Example
In this example,
- Master declares an array of tasks (
['task 1', 'task 2', 'task 3']
). createClusterMap
is used to create an instance ofclusterMap
.clusterMap
is used to task the workers.- Workers handle the tasks and reply to the master.
clusterMap
waits for all tasks to be processed.- When all tasks are processes,
clusterMap
resolves with an array of results.
;; if clusterisMaster let clusterMap tasks; tasks = 'task 1' 'task 2' 'task 3' ; clusterMap = ; ; if clusterisWorker ;
Using a separate file
master.js
;; let clusterMap tasks; tasks = 'task 1' 'task 2' 'task 3'; cluster; clusterMap = ; ;
worker.js
; ;
Logging
Logging is enabled using log
configuration.
Logging produces an output that describes:
- Number of forked processes.
- Logs time when worker is assigned a task.
- Logs time when worker responds with a result.
- Logs time when either of the tasks take longer than the
timeout
configuration to execute.
[03:36:46] Spawning 8 worker process(es).
[03:36:46] Tasking worker #1. "/bin/babel-external-helpers.js"
[03:36:46] Tasking worker #3. "/bin/babel-node.js"
[03:36:46] Tasking worker #5. "/bin/babel-plugin.js"
[03:36:46] Tasking worker #4. "/bin/babel.js"
[03:36:46] Tasking worker #7. "/index.js"
[03:36:46] Tasking worker #6. "/lib/_babel-node.js"
[03:36:46] Tasking worker #2. "/lib/babel-external-helpers.js"
[03:36:46] Tasking worker #8. "/lib/babel-node.js"
[03:36:48]
Received result from: /bin/babel-node.js
Tasks in progress (count): 7
Tasks in progress:
- /bin/babel-external-helpers.js
- /bin/babel-plugin.js
- /bin/babel.js
- /index.js
- /lib/_babel-node.js
- /lib/babel-external-helpers.js
- /lib/babel-node.js
Remaining tasks (count): 495
[03:36:48] Tasking worker #3. /lib/babel-plugin/index.js
[03:36:48]
Received result from: /bin/babel-external-helpers.js
Tasks in progress (count): 7
Tasks in progress:
- /bin/babel-plugin.js
- /bin/babel.js
- /index.js
- /lib/_babel-node.js
- /lib/babel-external-helpers.js
- /lib/babel-node.js
- /lib/babel-plugin/index.js
Remaining tasks (count): 494