jq for testers
As test automation engineers we often touch many different tools, protocols, data, and systems. Whatever we need to get the job done. Wrangling data can be quite burdensome, for example when preparing data for a next test or to verify expectations against data retrieved. Data can be in the wrong format, have additional unwanted fields, lack fields, or the receiving system expects a different structure.
Nowadays a lot of the data that we deal with is JSON. Simple enough to deal with and widely supported by tons of tools. It gets annoying when some of the issues above pop-up and you want to change the data for a lot of tests, or it’s just a lot of data.
Here is where jq can help. Jq is an open source command line tool, that makes it easy to deal with json in all kinds of scenarios. Below I’m sharing a few of the most useful applications I’ve come across for jq.
For the examples in this article I’m using data and endpoints from https://fakestoreapi.com/.
Pretty print & color json
Often systems return minified json, without white space and line breaks. While a bit more efficient to transmit, it makes it a lot harder to read for humans. Jq can help. Given a piece of json on your clipboard simply run:
pbpaste | jq
If you pass in JSON to the jq command it will automatically pretty-print and color the input. Nice!
Select only one field from an array of objects
https://fakestoreapi.com/users returns a list of users and their details. To grab only the address fields of each entry:
curl https://fakestoreapi.com/users | jq '.[].address'Slice the data
In order to grab the first 3 elements from an array you’d run:
curl https://fakestoreapi.com/users | jq '.[0:3]'Construct a new object
Sometimes you want to shuffle the data you have into another structure. jq again makes this easy.
curl https://fakestoreapi.com/users | jq '.[] | {user: "\(.name.firstname) \(.name.lastname)", phone: .phone}'
Count
Calling the length function gets you the number of items in an array or the number of key-value pairs in an object. Very useful if you want to run some quick verifications on the data.
curl https://fakestoreapi.com/users | jq lengthGet a list of all users whose e-mail address contains ‘w’
curl https://fakestoreapi.com/users|jq 'map(select(.email | contains("w")))'Count all users with a ‘w’ in their e-mail address
curl https://fakestoreapi.com/users|jq 'map(select(.email | contains("w")))' | jq lengthThere’s a lot more you can do with jq, check out the documentation for much, much more you can do with it.
