Continuous integration

Logging in from CI, frozen installs, publishing without prompts

Chapter 28 of 34Current for Gritpack 1.2.63Source: docs/guide/28-continuous-integration.md

A CI job is a person who cannot answer questions. It needs Gritpack installed, a login it can use without typing a code, exactly the packages the lock file names, and a way to publish a release without being asked "are you sure?". This chapter builds that job one step at a time. The examples use GitHub Actions syntax because it is common, but every step is a plain shell command, so the same recipe works in any CI system.

What you will be able to do

  • Install Gritpack in a fresh CI machine.
  • Give the job a login from your CI system's secrets, safely.
  • Install exactly the locked packages, and fail when the lock is stale.
  • Keep the download cache between runs.
  • Publish a release from CI and keep its signed receipt.

What changes when nobody is watching

Three things behave differently without a person at a terminal:

  • Gritpack never prompts. When standard input is not a terminal, or the CI variable is set, nothing waits for an answer. A command that would ask needs --yes, and fails with a usage error (exit 2) without it.
  • login cannot ask for a code. A one-time code is for a person. It works once and expires within a day, so it cannot live in a CI secret.
  • Output is read by programs. Ask for JSON results and JSON errors (see Scripting with JSON), so later steps read named fields rather than sentences.

Step 1: install Gritpack

On a Linux runner, the one-line installer from Install Gritpack works without a person if you pass --yes. Arguments after sh -s -- reach the installer:

curl -fsSL https://get.gritpack.org/install/gritpack | sh -s -- --yes --no-modify-path
echo "$HOME/.frogfish/gritpack/bin" >> "$GITHUB_PATH"
gritpack version

--no-modify-path stops the installer offering to edit a shell profile that the job will never read. The second line is how GitHub Actions adds a directory to PATH for the following steps; other CI systems have their own way. On a Windows runner, PowerShell's irm cannot pass arguments, so use the form that can:

& ([scriptblock]::Create((irm https://get.gritpack.org/install.ps1/gritpack))) --yes

For automation. gritpack version --json prints the version, platform and the path of the executable. Log it at the start of every job, so a failure can be matched to the release that produced it.

Step 2: a login for the job

A CI job should have its own account, not borrow a person's. Ask your hub's administrator to add one, for example ci-releases@example.com, and to give it only the scopes the job publishes to (see Administering a hub).

Then make the credential once, on a trusted machine, with the invitation code the administrator sends:

gritpack login ci-releases@example.com --hub packages.example.com \
  --code <code> --store file --profile ci

--store file keeps the credential as three files under ~/.frogfish/gritpack/credentials/<credential-id>/:

FileWhat it is
client.pemthe credential's certificate; not secret
client-key.pemits private key; secret
server-ca.pemthe certificate authority that vouches for the hub; not secret

Put the contents of those files into three CI secrets. In GitHub that is Settings, Secrets and variables, Actions, New repository secret; call them GRITPACK_CLIENT_CERT, GRITPACK_CLIENT_KEY and GRITPACK_HUB_CA. Then delete the copies on the trusted machine, or log that profile out once the secrets are saved.

In the job, write the secrets to files and save them as a profile with auth import:

- name: Gritpack login
  env:
    CLIENT_CERT: ${{ secrets.GRITPACK_CLIENT_CERT }}
    CLIENT_KEY: ${{ secrets.GRITPACK_CLIENT_KEY }}
    HUB_CA: ${{ secrets.GRITPACK_HUB_CA }}
  run: |
    umask 077
    mkdir -p "$RUNNER_TEMP/gritpack"
    printf '%s\n' "$CLIENT_CERT" > "$RUNNER_TEMP/gritpack/client.pem"
    printf '%s\n' "$CLIENT_KEY"  > "$RUNNER_TEMP/gritpack/client-key.pem"
    printf '%s\n' "$HUB_CA"      > "$RUNNER_TEMP/gritpack/server-ca.pem"
    gritpack auth import ci --hub packages.example.com \
      --cert "$RUNNER_TEMP/gritpack/client.pem" \
      --key "$RUNNER_TEMP/gritpack/client-key.pem" \
      --ca "$RUNNER_TEMP/gritpack/server-ca.pem" --default
    gritpack auth status

auth import saves a profile that names the files; it does not copy the key anywhere else. On a test run it looks like this:

$ gritpack auth import ci --hub packages.example.com --cert client.pem --key client-key.pem --ca server-ca.pem --default
saved profile ci in ~/.frogfish/gritpack/config.toml
$ gritpack auth list
ci (default) server=packages.example.com provider=file
$ gritpack auth status
account=01a1e865-dd2b-7f10-9340-1658dbafa174 email=ci-releases@example.com status=Active
...

Without a profile: environment variables

If you would rather not save a profile at all, every connection flag has an environment variable. With these four set, every command uses them:

env:
  GRITPACK_SERVER: packages.example.com
  GRITPACK_CERT: ${{ runner.temp }}/gritpack/client.pem
  GRITPACK_KEY: ${{ runner.temp }}/gritpack/client-key.pem
  GRITPACK_CA: ${{ runner.temp }}/gritpack/server-ca.pem

A flag on the command line still wins over the variable. GRITPACK_PROFILE chooses a profile the same way, but it cannot be combined with the other four.

When the credential runs out

A login lasts 30 days. From a week before it expires, every command that uses it prints a warning on standard error saying when it expires and that it should be renewed with gritpack auth renew. Make that warning visible: let the job's log show standard error, or check gritpack auth status --json and fail early when expires_at is close.

Renewing needs a new invitation code from the administrator, so it is a person's job. On the trusted machine:

gritpack auth renew --profile ci --hub packages.example.com --code <new code>

then copy the new files into the secrets. An expired login cannot renew itself; Gritpack says so and points to gritpack login.

Careful. The private key in GRITPACK_CLIENT_KEY can publish to every scope the CI account owns. Give the CI account no more scopes than it needs, keep the secret out of pull requests from forks, and ask the administrator to revoke it (admin user view shows its credentials) if it leaks.

Step 3: install exactly what the lock file says

In CI, use sync --frozen:

gritpack sync --frozen

--frozen installs the exact versions in gritpack.lock and fails if they would change. That catches the commit where someone edited gritpack.toml and forgot to commit the new lock:

$ gritpack sync --frozen
locked package @ada/textkit is unavailable or no longer matches its requirements

(Here gritpack.toml had been changed to require ^1.0.2 while the lock still named 1.0.0.)

The command exits with status 5 and changes nothing. The fix belongs on the developer's machine: run gritpack sync (or add, update) there and commit gritpack.lock. See The lock file and sync for what --frozen and --offline promise.

A locked package keeps its source: the lock records the hub each package came from. A CI machine must be able to reach those hubs, and its login must be for the same private hub the developers use.

Step 4: keep the download cache

Every downloaded package file is kept in a cache addressed by its SHA-256, under ~/.frogfish/gritpack/cache (or GRITPACK_CACHE_DIR). Every read from the cache checks the digest again, so restoring a cache from an earlier run is safe: a damaged file is fetched again, never used.

- uses: actions/cache@v4
  with:
    path: ~/.frogfish/gritpack/cache
    key: gritpack-${{ runner.os }}-${{ hashFiles('**/gritpack.lock') }}
    restore-keys: gritpack-${{ runner.os }}-

With a warm cache you can go further and install with no network at all:

gritpack sync --offline

--offline implies --frozen and reads only the lock file and the cache. If the cache lacks a file, it fails with "offline cache does not contain ..." (exit 5) instead of reaching for the network. That is useful in build steps that must not depend on the hub, and in air-gapped builders fed by a bundle (see Offline bundles and slices).

For automation. gritpack cache prune --max-bytes 2GiB --apply keeps a long-lived self-hosted runner's cache from growing without bound. Without --apply it only reports what it would remove.

Step 5: check, and publish a release

On every pull request, run the same checks a publication would, without publishing:

gritpack publish --dry-run

A dry run builds the package files, checks the README and licence rules, and asks the hub whether it would accept the versions. It never changes the hub. A failure here (a missing README, a licence the hub's rules refuse, a version someone already published) is the same failure the release would hit, found earlier.

On a release tag, publish for real. --yes answers the confirmation, and JSON gives the next step something to read:

- name: Publish
  run: |
    gritpack publish --yes --json > publish.json
    echo "transaction=$(jq -r .receipt.transaction_id publish.json)" >> "$GITHUB_OUTPUT"

The result's top-level fields are mutated, plan_path, receipt, schema (gritpack/publication-result@1) and verification. receipt.transaction_id names the publication; it is also the ID of its signed receipt. Keep that receipt with the build:

gritpack evidence receipt "$TRANSACTION" --output gritpack-receipt.json

Evidence and receipts explains what the receipt proves and how to check it later without the hub.

Retries are safe

Publication is a transaction: every package in the plan becomes visible together, or none does. If a job is cancelled or the network drops, run the same job again. Running publish again with the same plan finishes the publication that is already under way instead of starting a second one; when the transaction had already committed, Gritpack says so:

$ gritpack publish --yes --jq '.receipt.transaction_id'
publication transaction 01a1e867-3b33-7eb0-9d9d-ea97e5fe85db is already Committed; finishing its commit
...

What is never safe is changing the files and publishing the same version again. A version's content cannot change; the hub refuses with PUBLICATION_IDENTITY_CONFLICT and exit 6. Raise the version instead.

For a release that must be paused between planning and publishing (an approval step, a signature from another machine), use the step-by-step commands in Publishing step by step.

A complete workflow

Putting the steps together:

name: build
on:
  push:
    tags: ["v*"]
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      GRITPACK_ERROR_FORMAT: json
    steps:
      - uses: actions/checkout@v4

      - name: Install Gritpack
        run: |
          curl -fsSL https://get.gritpack.org/install/gritpack | sh -s -- --yes --no-modify-path
          echo "$HOME/.frogfish/gritpack/bin" >> "$GITHUB_PATH"

      - name: Gritpack login
        env:
          CLIENT_CERT: ${{ secrets.GRITPACK_CLIENT_CERT }}
          CLIENT_KEY: ${{ secrets.GRITPACK_CLIENT_KEY }}
          HUB_CA: ${{ secrets.GRITPACK_HUB_CA }}
        run: |
          umask 077
          mkdir -p "$RUNNER_TEMP/gritpack"
          printf '%s\n' "$CLIENT_CERT" > "$RUNNER_TEMP/gritpack/client.pem"
          printf '%s\n' "$CLIENT_KEY"  > "$RUNNER_TEMP/gritpack/client-key.pem"
          printf '%s\n' "$HUB_CA"      > "$RUNNER_TEMP/gritpack/server-ca.pem"
          gritpack auth import ci --hub packages.example.com \
            --cert "$RUNNER_TEMP/gritpack/client.pem" \
            --key "$RUNNER_TEMP/gritpack/client-key.pem" \
            --ca "$RUNNER_TEMP/gritpack/server-ca.pem" --default

      - uses: actions/cache@v4
        with:
          path: ~/.frogfish/gritpack/cache
          key: gritpack-${{ runner.os }}-${{ hashFiles('**/gritpack.lock') }}
          restore-keys: gritpack-${{ runner.os }}-

      - name: Install dependencies
        run: gritpack sync --frozen

      - name: Build and test
        run: make test

      - name: Check the release
        if: github.event_name == 'pull_request'
        run: gritpack publish --dry-run

      - name: Publish
        if: startsWith(github.ref, 'refs/tags/v')
        run: |
          gritpack publish --yes --json > publish.json
          gritpack evidence receipt "$(jq -r .receipt.transaction_id publish.json)" \
            --output gritpack-receipt.json

For Cargo, npm and Go projects the same job usually adds one more step after login, the setup command from that ecosystem's chapter, so the ecosystem's own tool can reach the hub. See Rust and Cargo, npm and Go modules.

The variables a CI job uses

VariableWhat it does
CIWhen set, Gritpack never prompts
GRITPACK_ERROR_FORMATjson makes every failure one JSON object on standard error
GRITPACK_SERVER, GRITPACK_CERT, GRITPACK_KEY, GRITPACK_CAThe hub and the credential, instead of a profile
GRITPACK_PROFILEWhich saved profile to use
GRITPACK_HOMEMoves the whole Gritpack directory, for example into the job's workspace
GRITPACK_CACHE_DIRMoves only the download cache
GRITPACK_JOBSHow many downloads run at once, 1 to 64 (default 4)
GRITPACK_PACKAGES_DIRWhere a project's packages are installed

gritpack help environment lists them all.

What just happened

You installed Gritpack on a machine with no person at it, gave it a login of its own from secrets, installed exactly what the lock file names, kept the cache between runs, checked releases on every pull request, and published on tags with a receipt you can keep. Nothing in the job waits for an answer, and every failure has a stable exit code.

You can stop here

Teams that build and release from CI have the whole recipe. If your CI only installs dependencies, steps 1 to 4 are all you need.

Next

Make the interactive shell more pleasant in Shell comforts. For releases of several packages at once, read Releasing a workspace.