express-stream-json

Middleware for express that streams a json array to the response stream.

Express normally serves json by stringifying an object and sending to the response in one chunk. This, of course, means that you have to have the entire object in memory before you can start the response.

This middleware is for the use case where one is using streams server side and don't want to consume that stream and create a huge object in memory on the server. It is especially useful when you are retrieving database records as an object stream and the stream gets a chunk for each row.

The middleware should follow routes and the stream must be attached to the request object as req.resultStream.

If req.resultStream is not present or is not a ReadableStream the middleware does nothing and just forwards to next.

The middleware assumes that the entire response represents an array and that each row in the array is a json string or a buffer representing the same. The middleware transforms the stream by prepending the first chunk with "[", adding a comma after each chunk except the last, and appending "]" after the last chunk is received.

The transformed stream is piped to res so the res can start streaming as soon as it gets the first row of data.

Installation

VERSION 4 is likely broken

npm install express-stream-json

Usage

In app.js add the following line after the routes section

use require("express-stream-json")

In the processing that your routes call attach the stream to be processed:

...
req.resultStream=MyStream
...
next();

If you are not using an array stream just use the normal method of sending results and do not create req.resultStream. The middleware will then do nothing.

res.send(results);
next()

Obviously the middleware is only useful for browsers which can handle chunked responses so it is of no use for IE8 and earlier.

Even if the client does not expose the results until the entire response is received using the middleware still offers the advantage of reduced memory and less latency in the the start of the response.