Streaming CSV Parser. Made entirely out of streams.


Parses csv files. Accepts options. No coffee script, no weird APIs. Just streams. Tested against csv-spectrum and used in production.

Works in node 0.10, 0.12 and iojs.

npm install csv-streamify

This module implements a simple node stream.Transform stream.

var csv = require('csv-streamify'),
    fs = require('fs')
var fstream = fs.createReadStream('/path/to/file'),
    parser = csv(options /* optional */, callback /* optional */)
// emits each line as a buffer or as a string representing an array of fields 
parser.on('readable', function () {
  var line = parser.read()
  // do stuff with data as it comes in 
  // current line number 
// AND/OR 
function callback(errdoc) {
  if (err) return handleErrorGracefully(err)
  // doc is an array of row arrays 
  doc.forEach(function (row) { console.log(row) })
// now pump some data into it (and pipe it somewhere else) 

Note: If you pass a callback to csv-streamify it will buffer the parsed data for you and pass it to the callback when it's done. This behaviour can obviously lead to out of memory errors with very large csv files.

You can pass some options to the parser. All of them are optional.

The options are also passed to the underlying transform stream, so you can pass in any standard node core stream options.

  delimiter: ',', // comma, semicolon, whatever 
  newline: '\n', // newline character (use \r\n for CRLF files) 
  quote: '"', // what's considered a quote 
  empty: '', // empty fields are replaced by this, 
  // if true, emit arrays instead of stringified arrays or buffers 
  objectMode: false,
  // if set to true, uses first row as keys -> [ { column1: value1, column2: value2 }, ...] 
  columns: false

Also, take a look at iconv-lite (npm install iconv-lite --save), it provides pure javascript streaming character encoding conversion.

To use on the command line install it globally:

$ npm install csv-streamify -g

This should add the csv-streamify command to your $PATH.

Then, you either pipe data into it or give it a filename:

# pipe data in
$ cat some_data.csv | csv-streamify
# pass a filename
$ csv-streamify some_data.csv > output.json
# tell csv-streamify to read from + wait on stdin
$ csv-streamify -
  • browser support

If you would like to contribute either of those just open an issue so we can discuss it further. :)

Nicolas Hery (objectMode)