Linux has no built in way to handle JSON properly, yet many APIs will return JSON output. Here’s how to parse and use JSON data inside your shell scripts with a simple utility.
Parsing JSON With jq
jq is a utility that can handle everything you may want to do with JSON. It’s super simple to install; simply download the binary and move it to your a folder on your PATH, usually /usr/local/bin/. It has no dependencies, so it’s just one binary to download. There are downloads for macOS and Windows as well.
To start, you can pipe JSON to it to pretty-print it. Some APIs will minify their responses to save space, leading to this awful mess when you curl something in the console:
Pipe it to jq like so:
And it’s now readable, with syntax highlighting to boot:
If you give jq a command, it will use it to select data out of the JSON, much like how sed operates on text. We’ll use this JSONPlaceholder as an example API to try out commands. For simplicity’s sake, we’ll leave out the curl command and pipe to keep everything readable. You can also curl to a file and pipe cat to jq, if you don’t want to see the curl output every time you test a new command.
A simple period ‘.‘ represents the data piped into jq. If you simply want to get a key from the object, you can do:
You can nest this as well; for example, ‘.data.geo.host’ would return a value farther down the list. Optionally, you can add a question mark for error checking like “.data.geo?.host“, which specifies that the property is optional.
Working with arrays is where jq gets interesting. You can use square brackets to get a specific array element. For example, you could get the name of the first entry with:
You can slice arrays using [2:5], and use the same question mark syntax for optional properties. Though keep in mind that if an array is empty, it doesn’t mean it’s an invalid property.
However, you’ll likely want to fetch a specific entry based on the data within it, not on the order of the array. To do that, pass the whole array to the select utility:
This filters the array to only include items that pass the test; in this case, does the name field contain a certain name? This can be any boolean expression, even arithmetic ones like select(.count >= 2).
Reformatting JSON
jq can construct JSON as well, which means you can use it to reformat JSON input. You can make objects simply by wrapping them in curly brackets, and pass jq statements as the values. jq will output the new object. For example:
Outputs nicely to the command line:
While this is useful for stripping out only the information you want, you can also construct new objects using jq commands.
For full reference on all of jq‘s syntax, you can consult its online manual.
Alternatives to jq
If you don’t want to use another program, you could use a few alternatives. The easiest method is to forego doing it in bash, and instead use a scripting language like Python or JavaScript (with node) to do the actual processing. You would instead pipe the JSON command output to a file:
And then load it into a variable within the script for use.
You could also use simple text selection utilities; for example, if you just need a specific key out of a response, you could use grep alongside a regular expression to select the value:
However, this can (and probably will) break very easily with any changes in the JSON. Passing in the users API returned the name of each user, but also the username (which grep matched as well), as well as the name of the company (which had the exact same key). Unless you plan to account for all of this, you should use something that is aware of the JSON’s structure, rather than treating it as text.
