map-parallel

Data Parallelism as map() on separate threads in Node.js

While Node.js excels at asynchronous I/O, it falters when a long-running, CPU-bound execution takes place. The event loop blocks, thus any requests are indefinitely delayed until they possibly timeout.

This is map-parallel. It allows operating on an Array in a similar fashion as Array.map, albeit in-parallel and on separate threads.

It works by tessellating the input array in chunks, equal to the amount of available physical CPU cores then fires-off Child Processes, in parallel, on each chunk.

Any reasonable OS assigns each child-process to a different CPU core, thus allowing for truly parallel processing on each chunk.

Read more on Data Parallelism

Installation

Ensure you have Node.js 7.6 + installed, then:

$ npm i --save  map-parallel

Usage

 
const mapParallel = require("map-parallel");
 
mapParallel.map(arr, threads, cb, base).then((result) => {
  console.log(result);
})
.catch((err) => {
  console.error(err);
})

Example

 
const os = require("os");
const mapParallel = require("map-parallel");
 
mapParallel.map([1, 5], os.cpus().length, (item, base) => {
  return {
    item: item,
    included: base.includes(item)
  }
}, [1, 2, 3, 4]).then((result) => {
  // logs: `[{ item: 1, included: true }, { item: 2, included: false }]`
  console.log(result);
})
.catch((err) => {
  // handle err
})
 

Gotchas

No parent scope access

You don't have access to the parent scope in the context of the callback:

 
// parent scope
const x = 3;
 
mapParallel.map([1, 5], os.cpus().length, (item, base) => {
  return item.includes(x); // x is undefined!
}, []).then((result) => {
  // handle result
})
.catch((err) => {
  // handle err
})

Instead, use the base object, 3rd parameter:

const x = 3;
 
mapParallel.map([1, 5], os.cpus().length, (item, base) => {
  // base = x = 3;  
  return item.includes(base);
}, x).then((result) => { // x provided as input here
  // handle result
})
.catch((err) => {
  // handle err
})

Serialization Overhead

Serializing large base objects between child processes is computationally expensive. If your input size is very large but the computation on each element is light, then this module might not be a good fit.

Tests

$ npm install -g mocha
$ npm test

Authors

• Nicholas Kyriakides, @nicholaswmin

License

The MIT License

Copyright (c) 2017 Nicholas Kyriakides, [e-mail]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.