json-inspect 0.1.6

histo

If you are processing a large number of JSON objects/arrays, then it may be useful to know what fields are present, what values they contain, and the frequency of both. Starting off with the help file

usage: json-inspect histo [-h] [-p PREFIX] [-c] paths [paths ...]

Generate a histogram based on values found using a JSON search expression

positional arguments:
  paths                 search paths to create histograms for, prefixed with
                        optional value from the --prefix option

optional arguments:
  -h, --help            show this help message and exit
  -p PREFIX, --prefix PREFIX
                        String to prefix all search-paths with
  -c, --conflate        Conflate non-empty responses to the same value

• The -p option will allow you to prefix all of your search paths. This is useful if you are performing multiple searches that have a common prefix for deeply nested JSON searches.

• The -c option will conflate your histogram to two values, __none__ and __some__. The first meaning that no key/value was found for a given lookup in your search path and the latter meaning that a value was found. This is useful if you only care about the frequency of presence.

A search path is a dot-delimited expression used for traversing JSON objects. An example to get us started is

[].*.user.demographic.regions.[].name

To start, note that search expressions contain 3 types of tokens

• [] - indicates an array. Each item in the array is collected and will be processed by the next token

• * - indicates an object in which all fields should be collected and will be processed by the next token

• TOKEN - a field-value of an object. It’s value will be collected and processed by the next token

For our example above, it would be satisfied by the following JSON document

[
 {
   "facebook": {
     "user": {
       "demographic": {
         "regions": [ {"name": "US"}, {"name": "Kentucky"}, {"name": "Louisville"} ]
       }
     }
   },
   "google": {
     "user": {
       "demographic": {
         "regions": [ {"name": "US"}, {"name": "Kentucky"}, {"name": "Highland Heights"} ]
       }
     }
   }
 },
 {
   "twitter": {
     "user": {
       "demographic": {
         "regions": [ {"name": "UK"}, {"name": "Wales"} ]
       }
     }
   }
 }
]

Running the histo sub-command, we would get output such as

cat test.json | json-inspect histo '[].*.user.demographic.regions.[].name'

[].*.user.demographic.regions.[].name:
Highland Heig   | #########################                          | (1)
US              | ################################################## | (2)
Louisville      | #########################                          | (1)
Kentucky        | ################################################## | (2)
UK              | #########################                          | (1)
Wales           | #########################                          | (1)

The bar-chart represents the number found relative to the max found with a total count of finds per-element in the rightmost column.

ext

The ext, extraction command, is used for pulling data out of JSON files. It supports the same prefix and search expressions as histo along with a few other options for value output.

$ json-inspect ext -h
usage: json-inspect ext [-h] [-p PREFIX] [-d DELIM] [-F] paths [paths ...]

Extract values from JSON using a JSON search expression

positional arguments:
  paths                 search paths to return values for

optional arguments:
  -h, --help            show this help message and exit
  -p PREFIX, --prefix PREFIX
                        String to prefix all search-paths with
  -d DELIM, --delimiter DELIM
                        String to delimit all results by
  -F, --flatten         Flatten array and object values. For objects, only the
                        values (not the keys) are retained in the falttened
                        values

Using the same input test JSON file from the histo command, we can see an example of output

$ cat test.json | json-inspect ext "[].*.user.demographic.regions.[].*"
Louisville,Kentucky,US,Highland Heights,Kentucky,US,Wales,UK

$ cat test.json | json-inspect ext -d '|' "[].*.user.demographic.regions.[].*"
Louisville|Kentucky|US|Highland Heights|Kentucky|US|Wales|UK

keys

The keys command is used for listing alls keys found within a JSON document. Flags can be provided to filter which keys are extracted.

$ json-inspect keys -h
usage: json-inspect keys [-h] [-n] [-o] [-p] [-e]

List all keys within the JSON document using a period-delimited notation
similar to search-paths

optional arguments:
  -h, --help            show this help message and exit
  -n, --exclude-null    Exclude keys that contain a null value
  -o, --exclude-empty-objects
                        Exclude keys that contain an empty object
  -p, --exclude-empty-primitives
                        Exclude keys that contain an empty primitive value
                        (zero and empty string)
  -e, --exclude-empty-array
                        Exclude keys that contain an empty array value

Adding a new lines to each top-level object in our test.json with

"null": null, "empty_object": {}, "empty_array": [], "empty_string": "", "empty_int": 0, "empty_float": 0.0,

We can make some sample calls such as

bash $ cat test.json | json-inspect keys facebook.null facebook.empty_object facebook.user.demographic.regions.name facebook.empty_float facebook.empty_array # ...

bash # filter all keys with empty values $ cat test.json | json-inspect keys -nope facebook.user.demographic.regions.name google.user.demographic.regions.name twitter.user.demographic.regions.name