nq
npm install -g node-fn-query
cat example.json | jq -c .[] | nq '({eyeColor}) => eyeColor' # all eye color results
cat example.json | jq -c .[] | nq --filter '({favoriteFruit}) => favoriteFruit === "banana"' # filtered objects
cat example.json | jq -c .[] | nq --reduce [] '(acc, {friends}) => acc.concat(friends)' | jq -c .[] | nq '_.get("id")' | sort | uniq # list of unique friend ids. _ is lodash/fp
jq
needs a friend
why? I know there are people out there who have learned how to write complex jq
programs, and are even later able to read the programs that they've written. jq
is an amazing, wonderful, life-saving tool, but because I only need it every few weeks, I have major problems remembering the syntax, and I'll often need to leave the shell to deal with more complex JSON. Leaving the shell when you're in the middle of writing a pipeline or exploring some data is a real shame!
I've finally given up hope of ever learning jq
properly, so instead I've started using jq
to munge data into a workable state, and then manipulating it with nodejs
functions. The core of the code to make that happen is simple:
#!/usr/bin/env node
const fn = eval([].slice.call(process.argv, 2).join(" "));
require("readline")
.createInterface({ input: process.stdin })
.on("line", (v) => console.log(fn(JSON.parse(v))));
Just those few lines of code stuck somewhere appropriate in your PATH can make it much easier to manipulate JSON! It's incredibly freeing to not need to look anything up or leave the shell when you have a JSON problem.
usage
nq --help
nq
only reads stdin
line by line, so you'll almost always need to use jq --compact
(or jq -c
) to deal with objects, or jq -c .[]
(to iterate through an array) to get data into an nq
compatible format.
The default mode is mapping over the input values: seq 1 100 | nq 'n => n * n'
, but you can change that to either reduce
the values or filter
the values.
-
--reduce
(-r
) allows you to reducestdin
to a single value. It takes a value as its second parameter:seq 1 100 | nq -r 0 '(sum, n) => sum + n' # returns the sum
seq 1 20 | nq -r [] '(acc, n) => acc.concat(n)' # sticks the input into an array
-
--filter
(-f
) allows to filterstdin
:seq 1 100 | nq -f '(n) => n % 2' # prints odd numbers
By default, nq
attempts to JSON.parse
the input values & JSON.stringify
the output values:
-
--string-input
(-i
): don'tJSON.parse
input lines. Useful if you want to use this to process strings -
--raw-output
(-o
): don'tJSON.stringify
output lines. Useful if you want to feed string values into another shell script
lodash is included, and by default functional-style lodash/fp
is available as _
. The more functional lodash style lends itself to writing nq '_.get(["key", 0])'
rather than the slightly more verbose nq '(v) => _.get(v, ["key", 0])'
.
-
not-fp
:_
will be regularlodash
rather thanlodash/fp
. -
this
will be a consistent object that starts as{}
. Usingthis
to store state is generally a sign that you should bereducing
OR using a more fully-fleshed tool instead.
examples
For more complex examples, let's take a look at some of the highest upvoted jq
stackoverflow questions and answer them using nq
.
Given an array of objects with locations, how do we filter them by one of their properties?
-
jq
:jq '.[] | select(.location=="Stockholm")'
-
nq
:nq -f '({location}) => location === "Stockholm"'
Given data with a shape of Array<{Id: string, Names: string[]}>
, filter Id
based on the presence of a string containing "data"
in the Names
array.
-
jq
:jq -r '. - map(select(.Names[] | contains ("data"))) | .[] .Id'
-
nq
:jq -c .[] | nq --filter '({Names}) => Names.every((n) => !n.includes("data"))' | nq -o '({Id}) => Id'
-
nq
with a lodash functional style:nq --filter '_.flow( _.get("Names"), _.find(n => !n.match(/data/)) )' | nq -o '_.get("Id")
)
Given data that looks like {users: Array<{first: string, last: string}>}
, output ${first} ${last}
-
jq
:jq -r '.users[] | "\(.first) \(.last)"'
-
nq
:jq -c .users[] | nq -o '({first, last}) => `${first} ${last}`'
Note that for all of these cases, jq
is much more terse, is required for nq
to even do anything at all, and if we were processing a significant amount of data, I'd expect jq
to be much faster. nq
is worse than jq
in every way except offering familiar nodejs
syntax, but familiar syntax is a big deal!