map-parallel

    0.0.3 • Public • Published

    map-parallel

    Data Parallelism as map() on separate threads in Node.js

    Build Status

    While Node.js excels at asynchronous I/O, it falters when a long-running, CPU-bound execution takes place. The event loop blocks, thus any requests are indefinitely delayed until they possibly timeout.

    This is map-parallel. It allows operating on an Array in a similar fashion as Array.map, albeit in-parallel and on separate threads.

    It works by tessellating the input array in chunks, equal to the amount of available physical CPU cores then fires-off Child Processes, in parallel, on each chunk.

    Any reasonable OS assigns each child-process to a different CPU core, thus allowing for truly parallel processing on each chunk.

    Read more on Data Parallelism

    Installation

    Ensure you have Node.js 7.6 + installed, then:

    $ npm i --save  map-parallel

    Usage

     
    const mapParallel = require("map-parallel");
     
    mapParallel.map(arr, threads, cb, base).then((result) => {
      console.log(result);
    })
    .catch((err) => {
      console.error(err);
    })

    Example

     
    const os = require("os");
    const mapParallel = require("map-parallel");
     
    mapParallel.map([1, 5], os.cpus().length, (item, base) => {
      return {
        item: item,
        included: base.includes(item)
      }
    }, [1, 2, 3, 4]).then((result) => {
      // logs: `[{ item: 1, included: true }, { item: 2, included: false }]`
      console.log(result);
    })
    .catch((err) => {
      // handle err
    })
     

    Gotchas

    No parent scope access

    You don't have access to the parent scope in the context of the callback:

     
    // parent scope
    const x = 3;
     
    mapParallel.map([1, 5], os.cpus().length, (item, base) => {
      return item.includes(x); // x is undefined!
    }, []).then((result) => {
      // handle result
    })
    .catch((err) => {
      // handle err
    })

    Instead, use the base object, 3rd parameter:

    const x = 3;
     
    mapParallel.map([1, 5], os.cpus().length, (item, base) => {
      // base = x = 3;  
      return item.includes(base);
    }, x).then((result) => { // x provided as input here
      // handle result
    })
    .catch((err) => {
      // handle err
    })

    Serialization Overhead

    Serializing large base objects between child processes is computationally expensive. If your input size is very large but the computation on each element is light, then this module might not be a good fit.

    Tests

    $ npm install -g mocha
    $ npm test

    Authors

    License

    The MIT License

    Copyright (c) 2017 Nicholas Kyriakides, nik.kyriakides@gmail.com

    Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

    Install

    npm i map-parallel

    DownloadsWeekly Downloads

    2

    Version

    0.0.3

    License

    MIT

    Last publish

    Collaborators

    • nicholaswmin