Newbie Paintball Master

    ndjson-apply

    1.2.5 • Public • Published

    ndjson-apply

    Apply a JS function to a stream of newline-delimited JSON.

    Features:

    • take the JS function to apply from a file
    • the function may return async results
    • preview the transformation results with the --diff option

    NPM License Node

    Summary

    Install

    npm i -g ndjson-apply

    How To

    Basic

    cat some_data.ndjson | ndjson-apply some_transform_function.js > some_data_transformed.ndjson
    # Which can also be written
    ndjson-apply some_transform_function.js < cat some_data.ndjson > some_data_transformed.ndjson

    where some_transform_function.js just needs to export a JS function

    // some_transform_function.js
    module.exports = doc => {
      doc.total = doc.a + doc.b
      if (doc.total % 2 === 0) {
        return doc
      } else {
        // returning null or undefined drops the entry
      }
    }

    Async

    That function can also be async:

    const getSomeExtraData = require('./path/to/get_some_extra_data')
    
    // some_async_transform_function.js
    module.exports = async doc => {
      doc.total = doc.a + doc.b
      if (doc.total % 2 === 0) {
        doc.extraData = await getSomeExtraData(doc)
        return doc
      } else {
        // returning null or undefined drops the entry
      }
    }

    Diff mode

    As a way to preview the results of your transformation, you can use the diff mode

    cat some_data.ndjson | ndjson-apply some_transform_function.js --diff

    which will display a colored diff of each line before and after transformation.

    For more readability, each line diff output is indented and on several lines.

    Filter mode

    Use the js function only to filter lines: lines returning true will be let through. No transformation will be applied.

    cat some_data.ndjson | ndjson-apply some_transform_function.js --filter

    Use sub-function

    Given a function_collection.js file like:

    // function_collection.js
    module.exports = {
      foo: (obj) => {
        obj.timestamp = Date.now()
        return obj
      },
      bar: (obj) => {
        obj.count += obj.count
        return obj
      }
    }

    You can use those subfunction by passing their key as an additional argument

    cat some_data.ndjson | ndjson-apply ./function_collection.js foo
    cat some_data.ndjson | ndjson-apply ./function_collection.js bar

    Pass additional arguments

    Any remaining argument will be passed to the function

    # Pass '123' as argument to the exported function
    cat some_data.ndjson | ndjson-apply ./function.js 123
    # Pass '123' as argument to the exported sub-function foo
    cat some_data.ndjson | ndjson-apply ./function_collection.js foo 123

    See also

    • jq is great to work with NDJSON: cat entries_array.json | jq '.[]' -cr > entries.ndjson
    • ndjson-cli#map

    Install

    npm i ndjson-apply

    DownloadsWeekly Downloads

    30

    Version

    1.2.5

    License

    MIT

    Unpacked Size

    9.79 kB

    Total Files

    7

    Last publish

    Collaborators

    • maxlath