@wmfs/smithereens

1.54.0 • Public • Published

smithereens

Tymly Package npm (scoped) CircleCI codecov CodeFactor Dependabot badge Commitizen friendly JavaScript Style Guide license

Smash CSV files into more manageable files based on column values

Install

$ npm install smithereens --save

Usage

const smithereens = require('smithereens')

smithereens(
  [
    '/some/input/csv/files/people.csv'

    // people.csv:
    //
    // personNo,firstName,LastName,personType,action
    // 10,"Lisa","Simpson","c","u"
    // 20,"Homer","Simpson","a","u"
    // 30,"Bart","Simpson","c","d"
    // 40,"Marge","Simpson","a","d"
    // 50,"Maggie","Simpson","c","x"
    // 60,"Grampa","Simpson","x","u"
    // 70,"Milhouse","Van Houten","c","u"

  ],
  {

    outputDirRootPath: '/some/output/dir',

    parser: {
      quote: '"',
      delimiter: ',',
      newline: '\n',
      skipFirstLine: true,
      trimWhitespace: true
    },

    dirSplits: [
      {
        columnIndex: 3,
        valueToDirMap: {
          'c': 'children',
          'a': 'adults'
        }
      }
    ],

    fileSplits: {
      columnIndex: 4,
      valueToFileMap: {
        'u': {
          filename: 'changes',
          outputColumns: [
            {name: 'person_no', columnIndex: 0},
            {name: 'first_name', columnIndex: 1},
            {name: 'last_name', columnIndex: 2}
          ]
        },
        'd': {
          filename: 'deletes',
          outputColumns: [
            {name: 'person_no', columnIndex: 0}
          ]
        }
      }
    }
  },

  function (err, manifest) {

    // File output
    // -----------
    //   /some/output/dir
    //   ./adults
    //     changes.csv:
    //       person_no,first_name,last_name
    //       20,Homer,Simpson
    //     deletes.csv:
    //       person_no
    //       40
    //   ./children
    //     changes.csv:
    //       person_no,first_name,last_name
    //       10,Lisa,Simpson
    //       70,Milhouse,Van Houten
    //     deletes.csv:
    //       person_no
    //       30
    //     unknown.csv:
    //       50,Maggie,Simpson,c,x
    //   ./unknown
    //     changes.csv:
    //       person_no,first_name,last_name
    //       60,Grampa,Simpson

  }
)

smithereens(sourceFilePaths, options, callback)

Arg Type Description
sourceFilePaths string | [string] A string or an array of strings identifying one or more files. Uses glob so /some/dir/*.csv style patterns are supported, as is directory recursion via /some/dir/**/*.csv
options object An object configuring how output should be produced. See Options for more information.
callback function To be of the form function(err, manifest). Manifest contains a summary of the output files produced.

Options

Property Type Description
outputDirRootPath string An absolute directory path where to write output to. All missing directories will be created.
parser object An parser object for configuring how input CSV files should be parsed.
dirSplits [object] An array of of dirSplit objects
fileSplits object A fileSplit object

parser object

Configures how to parse incoming CSV lines. Uses csv-streamify under the bonnet.

Property Type Description
skipFirstLine boolean Should the first line of each file be ignored? Set to true if files include a header line, for example.
delimiter string Comma, semicolon, whatever - defaults to comma.
newline string Newline character (use \r\n for CRLF files).
quote string What's considered a quote.
empty string Empty fields are replaced by this value.

dirSplit object

Smithereens can break CSV files across a nested set of directories based on values defined in each line.

Property Type Description
columnIndex integer Each line of each CSV file will be parsed into an array of strings. This value identifies which value to split on.
valueToDirMap object A simple mapping of an expected string value (as identified by columnIndex) and the directory name that this line should be routed to.

fileSplit object

In a similar way, Smithereens can route lines to different files, based on the contents of a parsed CSV column.

Property Type Description
columnIndex integer Identifies which of the parsed string values from each CSV line should be used to determine a filename that a row should be routed to.
valueToFileMap object A key/value map where key is a string value that is expected via columnIndex and value is a file object.

file object

Defines which filename a CSV row should be routed to, along with some output-formatting configuration.

Property Type Description
filename string The filename which a row should be routed to. All output files will be in CSV format. Note that the .csv extension is added automatically, so don't include it here.
outputColumns [object] An array of outputColumn objects - each defining a column that should appear in the output file.

outputColumn object

Defines the values for each column in the output.

Property Type Description
name string The name of the column, used in the first line of the CSV output
columnIndex integer Identifies the column in the corresponding row of the parsed incoming CSV array to copy to the output. If columnIndex is given any type parameter is ignored.
type string Describes other output - hash, uuid, constant. hash generates a hash value from the contents of the corresponding input row, uuid outputs a unique id, constant output a fix value, given by the value property.
value number or string A fixed value to put output when the type property is constant.

Testing

$ npm test

License

MIT

Readme

Keywords

Package Sidebar

Install

npm i @wmfs/smithereens

Weekly Downloads

92

Version

1.54.0

License

MIT

Unpacked Size

71.9 kB

Total Files

16

Last publish

Collaborators

  • wmfsbot