Scripting with JSON

--json, --json=fields, --jq, NDJSON, JSON errors and exit codes

Chapter 27 of 34Current for Gritpack 1.2.63Source: docs/guide/27-scripting-with-json.md

Sooner or later you want a script to read what Gritpack says: the newest version of a package, the transaction a release committed, whether a sync changed anything. Human output is written for people and may change between releases. JSON output keeps its schema. This chapter shows how to ask for it, how to cut it down to the fields you need, and how to tell one kind of failure from another without reading the error message.

What you will be able to do

  • Ask any command that prints a result for JSON, or for only some fields.
  • Run a jq expression over the result without installing jq.
  • Read streaming NDJSON from search and publish.
  • Get failures as one JSON object on standard error, and branch on the exit code.
  • Know which output is a stable contract and which is not.

Two streams: results and messages

Gritpack writes the result of a command to standard output and everything else (progress, notes, warnings, errors) to standard error. A pipe therefore receives only the result. You saw this already with publish: the line planning publication; ... is a message, so it never reaches the next program in a pipe.

That rule holds in human and JSON mode alike. It is why the commands in this chapter can be piped into other tools without filtering out chatter first.

Asking for JSON

Every command that prints a result has --format. Where JSON is offered, --json is a short way to write --format json:

$ gritpack search textkit
@ada/textkit 1.0.0 ecosystem=native score=900 dialect=c/c23 variants=1 — Small text helpers for C programs
$ gritpack search textkit --json
{"index_generation":"37db61c5...","next_cursor":null,"ranking_profile":"gritpack-lexical-v1","results":[{"deprecation_message":null,"description":"Small text helpers for C programs","ecosystem":"native","keywords":["example","text"],"latest_version":"1.0.0","matched":[{"field":"local_name","term":"textkit","weight":900}],"name":"@ada/textkit","primary_dialect":"c/c23","publisher":{"scope":"@ada"},"score":900,"status":"available","variant_count":1,"versions":["1.0.0"]}],"schema":"gritpack-catalogue-search/1"}

Most JSON documents carry a schema (or schema_version) field that names their shape and version, such as gritpack-catalogue-search/1 above or gritpack-project-change/1 for the project commands. A new field may appear in a later release; a field your script reads keeps its meaning while the schema name stays the same. Read fields by name and ignore the ones you do not know.

The project commands add, remove, sync, update and upgrade all print the same kind of report, so one script can read any of them:

$ gritpack sync --json
{
  "changed": false,
  "command": "sync",
  "packages": [
    {
      "dialect": "c/c23",
      "direct": true,
      "name": "@ada/greet",
      "source": "https://packages.example.com:443",
      "version": "0.2.0"
    }
  ],
  "packages_dir": "./packages",
  "schema": "gritpack-project-change/1"
}

changed tells you whether the packages directory was rewritten. A CI job can use it to decide whether a later step has anything to do.

For automation. gritpack help formatting prints a short summary of everything in this chapter. It is the same text in every installed copy of Gritpack, so it is a handy reminder on a machine you do not know.

Keeping only the fields you need

--json=a,b keeps only those fields of each record. When a command returns a list inside an envelope, as search does with results, the fields are taken from each record in that list:

$ gritpack search textkit --json=name,latest_version
[{"latest_version":"1.0.0","name":"@ada/textkit"}]

The field names are the names in the full JSON. If none of the names you give exists, Gritpack lists the ones that do and exits with status 2:

$ gritpack search textkit --json=nope
Error: unknown field nope; this command's fields are: deprecation_message, description, ecosystem, keywords, latest_version, matched, name, primary_dialect, publisher, score, status, variant_count, versions

That error is the quickest way to discover a command's fields. Be careful with a list that mixes known and unknown names: in 1.2.63 --json=name,nope keeps name and prints nope as null instead of failing. Check the spelling of each field once against the full --json output.

Note the =: --json name (with a space) means plain --json followed by an argument called name. That is how --json behaved before field selection existed, and it still does.

Running jq without jq

--jq EXPRESSION runs a jq expression over the JSON and prints the result. Gritpack carries its own jq engine, so nothing else needs to be installed. Strings are printed bare, one per line, which is what you usually want in a shell:

$ gritpack search --jq '.results[].name'
@ada/greet
@ada/textkit
$ gritpack sync --jq '.packages[] | .name + "@" + .version'
@ada/textkit@1.0.0

A few expressions that come up again and again:

# The versions of a package the hub offers, newest first
gritpack info @ada/textkit --jq '.families[0].family.publications[].version'

# Every direct dependency in the lock file
gritpack list --jq '.packages[] | select(.direct) | .name'

# Dependencies that have a newer version within their requirement
gritpack outdated --jq '.packages[] | select(.update_available) | .name'

# The transaction ID of a publication
gritpack publish --yes --jq '.receipt.transaction_id'

--jq implies JSON output, so you do not need --json as well. If the expression is not valid jq, the command stops with a usage error (exit 2):

$ gritpack search --jq '.results[[['
error: --jq: ".results[[[" is not a jq expression

Under the hood. info returns one entry in families per ecosystem that has a package with that name, so .families[0] is the first of them. Add --ecosystem native (or cargo) when a name exists in more than one ecosystem and you need a particular one.

Streaming with NDJSON

Two commands can stream: search and publish (and publication plan) take --format ndjson. Each record is one JSON object on its own line, so a program can act on each as it arrives:

$ gritpack search textkit --format ndjson
{"index_generation":"37db61c5...","ranking_profile":"gritpack-lexical-v1","schema":"gritpack-catalogue-search/1","type":"search_metadata"}
{"result":{"deprecation_message":null,"description":"Small text helpers for C programs","ecosystem":"native","keywords":["example","text"],"latest_version":"1.0.0","name":"@ada/textkit",...},"type":"package"}

Every line has a type, so a reader can switch on it. For publish, the plan records ("type":"plan", then one "type":"node" per package, then observations) stream as single lines.

Careful. In 1.2.63, publish --format ndjson prints the later records (the hub's verification report and the receipt) as indented JSON spread over several lines, not one per line. A strict NDJSON reader stops there. Until that is fixed, use --format json (one document at the end) when you need the receipt.

JSON errors

By default, a failure is a sentence on standard error. With --error-format json, or GRITPACK_ERROR_FORMAT=json in the environment, it is one JSON object instead, followed by a newline:

$ gritpack info no-such-package --error-format json
{"schema":"gritpack/error@1","code":"GRITPACK_NOT_FOUND","category":"not_found","exit_code":5,"message":"package no-such-package was not found","causes":[]}

--error-format affects only failures; a successful command prints exactly what it printed before. It works anywhere on the command line, and it covers mistakes in the arguments themselves:

$ gritpack search --frobnicate --error-format json
{"schema":"gritpack/error@1","code":"GRITPACK_USAGE","category":"usage","exit_code":2,"message":"error: unexpected argument '--frobnicate' found\n\n  tip: a similar argument exists: '--from-ca'\n\n...","causes":[]}

The fields schema, code, category and exit_code are the stable part. message and causes are for people: they explain, and their wording may change. causes is often the most useful part when you are debugging, because the human error shows only the first line. A hub whose certificate Gritpack does not trust, for example, reports only that the request failed in human mode, while the JSON names the reason:

$ gritpack login ada@example.com --hub packages.example.com --code AAAA-BBBB-CCCC-DDDD-EEEE-FFFF --store file --error-format json
{"schema":"gritpack/error@1","code":"GRITPACK_UNAVAILABLE","category":"unavailable","exit_code":8,"message":"enrollment request failed: error sending request for url (https://packages.example.com/v1/enrollments)","causes":["error sending request for url (https://packages.example.com/v1/enrollments)","client error (Connect)","invalid peer certificate: UnknownIssuer"]}

Exit codes

The exit code says what kind of failure happened. The list is short on purpose, and it is the same in every command:

ExitCodeWhat it meansExample that produced it
0Success
2GRITPACK_USAGEWrong arguments, configuration or requestan unknown flag; --profile naming a profile that does not exist; publish without --yes and no terminal
3GRITPACK_AUTHENTICATIONThe hub did not accept who you arelogin with a used, revoked or wrong code
4GRITPACK_AUTHORIZATIONYou are known, but may not do thisyanking a version in someone else's scope; any command from a suspended account
5GRITPACK_NOT_FOUNDNo such package, version, login or installed iteminfo no-such-package; add of a yanked exact version
6GRITPACK_CONFLICTSomething already there disagreesclaiming a scope someone owns; publishing a version again with different content; a publication blocked by the README rule or a licence rule
7GRITPACK_INTEGRITYA package, lock file or answer failed its checksa damaged file in the download cache; verify or doctor finding a missing installed file
8GRITPACK_UNAVAILABLEThe hub or network could not be reached; try againa hub that is down; a hub whose TLS certificate is not trusted
9GRITPACK_LOCAL_IOA local file or directory could not be read or writtenpack --output into a directory you cannot write
70GRITPACK_INTERNALGritpack could not finish safelyreport it

gritpack help exit-codes prints the same table. Scripts should branch on the exit code or on code, keep message for the log, and treat a code they do not know as a failure.

A small example in shell:

gritpack sync --frozen --error-format json 2> sync-error.json
case $? in
  0) echo "installed" ;;
  8) echo "hub unreachable; retrying later"; exit 75 ;;
  *) jq -r .message sync-error.json; exit 1 ;;
esac

For automation. Exit 8 is the only code that means "try again later" by itself. Everything else needs a change: different arguments, a different login, a fixed manifest or a repaired cache.

Prompts, and how scripts avoid them

Gritpack never prompts when standard input is not a terminal or when the CI environment variable is set. A command that would ask takes --yes instead:

$ gritpack publish
...
publication requires --yes when input is non-interactive or structured output is selected

publish asks for --yes whenever it cannot ask, and also whenever you ask for structured output, so a JSON-reading script never sits waiting on a question. The administrator commands that remove or hide something (admin user suspend and delete, admin scope unassign, admin package quarantine) and auth logout take --yes (or -y) too. In 1.2.63 they go ahead with a warning when there is nobody to ask; the warning says that from 1.4.0 they will need --yes. Pass it now and your scripts will not break.

Naming things the way people do

Scripts used to need storage IDs and account UUIDs. Since 1.2.62 you can name a published version as name@version and an account by its email wherever an ID was asked for:

gritpack package yank @ada/textkit@1.0.0
gritpack evidence attestations @ada/greet@0.1.0
gritpack admin user view ada@example.com

The IDs still work, and JSON output still includes them, so a script can store the ID it was given and use it later.

What just happened

You asked Gritpack for JSON, cut it down to the fields you needed, ran jq expressions without installing jq, read the streaming form, and turned failures into one machine-readable object with a stable exit code. Results went to standard output and everything else to standard error, so each pipe carried only what it should.

You can stop here

If you write shell scripts or small tools around Gritpack, you have what you need: ask for JSON, read named fields, and branch on the exit code.

Next

Put this to work in a pipeline in Continuous integration. If you would rather make the interactive shell faster, see Shell comforts.