multi-tasks

multi-tasks

Install:

npm install multi-tasks

How to use:

//see examples/example0
let multiTasks = require('multi-tasks').multiTasks;
//You need to provide the data of all subtasks, multi-tasks will automatically split them and execute them in a multi-threaded manner
let alltasks = [];
for(let i=0;i<100;i++){
    let task_props = {
        index: i,
        name: `task-${i}`,
        description: `This prop is for a subtask`
    };
    alltasks.push(task_props);
}

//You need to provide a process function to handle a certain sub-task and return the result data
let processTask = (task)=>{
    console.log('this is the subtask data you created above:', task);
    let {index} = task;

    //any exceptions will be captured
    if(index===3) throw 'exception';
    if(index===4) return Promise.reject({err:'a test error'});

    //the returned data will be saved
    return Promise.resolve({
        message: 'this is a result from a demo, random data=' + Math.random()
    });
}

multiTasks({
    tasks: alltasks, 
    processTask, 
    multi_task_parent_folder: `C:/tmp-demo`, //a directory to store progress and results files, you can check the progress here
    quotaOfEachTask:1, //The number of subtasks assigned to each worker at one time, the default is 1
    numberOfWorkers: 6, //Assign how many workers are working in parallel, default is the number of cpu cores minus one
    onFinish: (report)=>{
        console.log(report);//the report can be viewed here
    }
});

Or the older way, to provide the Producer & Consumer

let master = require('multi-tasks').master;
let RunnerProducer;//See examples, please provide a Producer class to specify all tasks you have, multi-tasks will automatically split them and execute them in a multi-threaded manner
let RunnerConsumer;//See examples, please provide a Consumer class to process a certain sub-task and return the processed data

master.start(RunnerProducer, RunnerConsumer, {
    multi_task_parent_folder,
    quotaOfEachTask,
    numberOfWorkers,
    onFinish
});

Have a try:

node examples/example0/run
node examples/example1/run

Changelog:

• 1.2.8 Fix: create task folder failed on MacOS
• 1.2.7 Small updates
• 1.2.6 Support onFinish event
• 1.2.5 Rename numberOfWorks to numberOfWorkers, the old one are still supported ;-)
• 1.2.4 Fix: opt.numberOfWorkers not work
• 1.2.3 Update README
• 1.2.2 Update README and examples
• 1.2.1 Handle exceptions and errors in subtasks
• 1.2.0 Simplified usage by providing the function way and support return Promise
• 1.1.4 Remove make-dir
• 1.1.3 Simplified usage, see example0
• 1.1.2
• 1.1.1 Rename files, updated changelog
• 1.1.0 Simplified the usage of a customized Consumer, see example0
• 1.0.8 Fix examples
• 1.0.7 Remove moment
• 1.0.6 Performance optimization

Github:

https://github.com/zhanglei923/multi-tasks