Sometimes you want to call an async function inside a transform stream and be able to take full advantage of the magical asynchronicity of node.js.
In order to do this, we must decouple the
this.push(data) from the
This module allows you to do that in a clean and simple way. See below.
var parallelStream = ;var maxInFlight = 2;var seeYouLaterStream =;
NOTE: the end function is optional
Once you've made your new parallel stream, you can use it just like you would any normal transform stream.
Just throw it into a
.pipe() call, like so.
var streamArray = ;var sink = ;// <--- there it is;
You can see the full example here
Now let's play around with this
Let's first see how setting the
1 results in a strandard serial transform stream.
So the output will look like this...
$ time npm run example -- 1I see you, DianaOh hey there again, DianaI see you, JulianOh hey there again, JulianI see you, StephenOh hey there again, StephenI see you, PeterOh hey there again, Peterreal 0m4.256suser 0m0.114ssys 0m0.021s
Now let's set it to
2 and see how different the output looks, and if performance has improved.
$ time npm run example -- 2I see you, DianaI see you, JulianOh hey there again, DianaI see you, StephenOh hey there again, JulianI see you, PeterOh hey there again, StephenOh hey there again, Peterreal 0m2.258suser 0m0.128ssys 0m0.025s
You can see when we allow 2 requests in flight, we get the first 2 requests back-to-back, send then off for async
handling and then pause to wait for one of them to return and make room for the next incoming request.
As soon as we've seen one of the first 2 requests come back (
Oh hey there again, Diana),
another incoming requests comes in (
I see you, Stephen). And let's note that the amount of time it took to get through
all the data has been cut in half, because... asynchronous!
Finally, let's run it with
maxInFlight set to
200, which is just a number larger than the length of the input data array.
$ time npm run example -- 200I see you, DianaI see you, JulianI see you, StephenI see you, PeterOh hey there again, DianaOh hey there again, JulianOh hey there again, StephenOh hey there again, Peterreal 0m1.159suser 0m0.121ssys 0m0.022s
You can see that all the requests were sent out at once, and all the responses came in shortly thereafter. Note how quickly it all happened, too.
We rely on semantic-release and Greenkeeper to maintain our module and dependency versions.