Releasing

A release goes from a release PR to a published GitHub release in seven steps. CI validates the signed tag and creates a draft release; it never builds anything. The maintainer builds on the build server, signs on the laptop, and publishes from the build server once every check passes. Each step is a make target that calls etc/release.sh. release-tag, release-sign and release-publish show what they are about to do and wait for you to type yes. The examples use v0.17.0-adiri.

At a glance

Release channels

release-build pushes the image under the tag name. release-publish moves :latest or :adiri only when the release is the highest published final version in its channel, so publishing a lower version after a higher one does not move the alias back. It marks a release Latest only when it is a mainnet final release with the highest published mainnet version, and every release candidate and Adiri release a pre-release. The version rules below refuse a final version that is not higher than every final version already tagged in its channel, so a patch to an earlier minor version, such as v1.1.1 once v1.2.0 exists, cannot be released. A mainnet final is published as a normal release that is not Latest only when a higher version, tagged after it, was published first. etc/release.sh rejects any tag outside this grammar, so neither CI nor the make targets accept one; numbers have no leading zeros, and N starts at 1.

The tag checks also enforce these version rules:

• the release commit sets [workspace.package].version in Cargo.toml to X.Y.Z, with no suffix;
• a final version must be higher than every other final version in the same channel;
• a release candidate’s version must be higher than every final version in the same channel, so none can be tagged once its own final release exists;
• mainnet and Adiri may use the same X.Y.Z;
• telcoin-network/vX.Y.Z/linux must fit in the 32-byte block extra-data field.

A tag is validated against every release tag that exists when it is pushed. release-prep and release-tag apply the version rules to the tags on GitHub, read with git ls-remote, because a clone’s own tags can be out of date. When the tag is pushed, CI compares it with every v* tag in its checkout; a re-run of that check compares it with the tags created before GitHub created the run, the time of the push. Every later step fetches the v* tags from origin, and stops when a local one differs from origin’s, then compares with the tags dated before the tag’s tagger date, on any commit; that date must not be earlier than the release commit’s committer date or more than five minutes ahead of the host’s clock. So an older release still verifies after newer ones exist, while a lower version tagged after a higher one, or a release candidate tagged after its final release, fails.

Machines and prerequisites

Laptop (macOS)

• YubiKey signing setup is done, and your key is on the allowlist on main.
• gpg --card-status shows the YubiKey.
• git config --get user.signingkey prints your signing key fingerprint followed by !; RELEASE_GPG_KEY overrides it for one command.
• gh auth status shows a login with the repo scope, which release-sign needs to read the draft and upload to it.
• Docker runs linux/amd64 images, as Docker Desktop does on any Mac: release-sign pulls the release image by digest, compares its binary with the tarball’s and runs its --version.
• The repository is cloned and on an up-to-date main.
• Foundry’s cast is optional: with it, release-tag and release-sign also check the on-chain attestation, and without it they print a notice and leave that check to CI.

Build server (Ubuntu or Debian, Linux x86_64)

Any Ubuntu or Debian server on Linux x86_64 can be the build server; release-build refuses any other platform.

• docker buildx version works, and docker buildx inspect default reports the docker driver; RELEASE_BUILDER selects another builder with that driver.
• gh auth status shows a login with the repo scope, to upload to the draft and edit it.
• Docker is logged in to ghcr.io through make docker-login; Registry access on the build server covers the first login.
• cast --version works, and .env holds GITHUB_ATTESTATION_PRIVATE_KEY for an address with the MAINTAINER role (see the CI environment notes).
• make init-submodules has been run.
• The checkout is on an up-to-date main with a clean tracked tree. release-prep requires HEAD to be origin/main. release-build and release-publish refuse to run when etc/release.sh or .github/scripts/verify_commit_hash.sh differ from origin/main.

The YubiKey never goes to the build server, and the registry token never goes on the laptop.

One-time setup

• The allowlist on main must hold a provisioned key; every check fails on a placeholder file and names it.
• A repository admin sets up the tag ruleset on v*, as Settings the release workflow requires describes: only maintainers can create a v* tag, and it must be signed; nobody, maintainers included, can update or delete one. The scripts cannot see the ruleset, and the signatures do not depend on it: every step that reads a tag requires exactly one signature on it, made by a key in the allowlist on main, and publishing requires at least MIN_SIGNATURES maintainer signatures over SHA256SUMS. The ruleset covers what signatures cannot: GitHub runs release.yaml from the tagged commit, so the ruleset decides who can start the workflow for a release tag, and it keeps a tag from being moved or deleted after it was validated.
• The ghcr package must be public, because the release scripts and operators read it without credentials. Push any image to it first (for example with make docker-adiri), then set its visibility to public at https://github.com/orgs/Telcoin-Association/packages/container/telcoin-network/settings. Until then, release-build stops with package is not public.

Cut a release

1. Prepare and merge the release PR (build server)

git switch main && git pull --ff-only
make release-prep TAG=v0.17.0-adiri

release-prep checks that the tracked tree is clean, that HEAD is origin/main, and that the tag exists neither locally nor on GitHub. It sets [workspace.package].version to 0.17.0, runs cargo update --workspace, and generates the new CHANGELOG.md section with git-cliff in Docker. The section records the main commit it was generated on in a <!-- release-base: <sha> --> line under its heading. It prints the section and a Next: list that starts with these commands:

git switch -c release/v0.17.0-adiri
git commit -am "release: v0.17.0-adiri"
gh pr create --title "release: v0.17.0-adiri" --body "Version 0.17.0 and the CHANGELOG section for v0.17.0-adiri."

Review the section, but do not edit it; CHANGELOG.md sections are generated, never written by hand. Notes for operators, such as a required resync, a configuration change or an activation epoch, go in the GitHub release in step 4.

Attest the PR head with make attest as for any PR, then merge it through the merge queue on its own. The tag checks require the release commit’s parent to be the release-base commit, so the PR must land alone and before anything else reaches main. If main moves before the PR merges, close it and repeat this step on the new main. If the release commit landed in a batch with another PR, repeat this step on the new main; release-prep replaces the section it wrote earlier, since that section is still the newest one.

When the PR has merged, attest the release commit, which is the commit you will tag:

git switch main && git pull --ff-only
RELEASE_SHA="$(git log -1 --format=%H --grep='^release: v0\.17\.0-adiri (#' origin/main)"
git switch --detach "$RELEASE_SHA" && ALLOW_STALE_BASE=1 make attest
git switch main

RELEASE_SHA is the newest commit on main whose subject is release: v0.17.0-adiri (#<PR>), which is the release commit you just merged; it is the tip of main unless another PR has landed since. The pattern ends at (#, which starts the pull request number of the squash commit, so it matches neither a candidate’s commit, release: v0.17.0-adiri-rc1 (#<PR>), nor, for a mainnet tag such as v0.17.0, the Adiri commit of the same version, release: v0.17.0-adiri (#<PR>). make attest normally refuses a commit that is behind origin/main, because a PR attested on an old base is not what the merge queue merges. The release commit is already on main and is the exact commit that gets tagged and built, so ALLOW_STALE_BASE=1 is safe here. If make attest reports that the commit is already attested, there is nothing more to do. Switch back to main afterwards, because release-build and release-publish compare the release scripts in the checkout with origin/main.

The new section shows up on the docs site’s Release notes as soon as the PR merges.

2. Sign and push the tag (laptop)

git switch main && git pull --ff-only
make release-tag TAG=v0.17.0-adiri

release-tag tags the tip of origin/main. If another PR has landed since the release commit, point it at the release commit instead:

RELEASE_COMMIT="$(git log -1 -E --format=%H --grep='^release: v0\.17\.0-adiri(-rc[1-9][0-9]*)? \(#' origin/main)" make release-tag TAG=v0.17.0-adiri

This pattern also matches a release candidate’s commit of the same version and channel, such as release: v0.17.0-adiri-rc1 (#<PR>), and -1 picks the newest, so it finds the candidate’s commit when the final release tags the same commit. It ends at (# as well, so for a mainnet tag it never matches the Adiri commit of the same version. The release commit must already be attested, as in step 1.

Before signing, it checks the Cargo version and the CHANGELOG.md section on that commit, and the attestation too when cast is installed. It signs with RELEASE_GPG_KEY if set and git config user.signingkey otherwise, and refuses a key whose primary key is not on the allowlist. The tag message is Release v0.17.0-adiri. It verifies the new tag the way CI will, shows what it is about to push, and pushes the tag after you type yes. Expect one touch, and a PIN prompt if this is the first signature since the YubiKey was plugged in.

3. CI validates the tag and creates the draft

Pushing the tag starts release.yaml. Its validate-tag job runs etc/release.sh check-tag with the scripts and the allowlist from main, and checks that:

• the tag is annotated and carries exactly one signature, made by an allowlisted key;
• the tagged commit is on main;
• Cargo.toml has the tag’s version and the version rules hold;
• CHANGELOG.md has the release’s section with the right release-base;
• the commit is attested on-chain.

The job writes the generated release notes to its summary. The draft-release job then creates a draft release with those notes. release-build refuses a draft that github-actions[bot] did not create, which catches a draft made by hand. This is not a security check, because anyone with write access can edit a draft or create one from a workflow of their own, so the later steps re-check the tag, the assets and the signatures instead of trusting the draft.

gh run list --workflow release.yaml --limit 3
gh run watch
gh release view v0.17.0-adiri

4. Build (build server)

git switch main && git pull --ff-only
make release-build TAG=v0.17.0-adiri

release-build checks the host, the ghcr login, that the release scripts match origin/main, every tag check from step 3, that the draft exists and has no signature yet, and that :v0.17.0-adiri is not in the registry yet. It builds the linux/amd64 image from a temporary worktree of the tag, without the build cache, and copies the binary out of the image. It runs --version in the image as a smoke test, packs the tarball, and pushes the image as ghcr.io/telcoin-association/telcoin-network:v0.17.0-adiri. It writes IMAGE_DIGEST and SHA256SUMS next to the tarball in target/release-artifacts/v0.17.0-adiri/, uploads the three files to the draft, and rewrites the draft’s notes to include the image digest. SHA256SUMS lists IMAGE_DIGEST, the tarball, and the network’s committee.yaml, genesis.yaml and parameters.yaml by their repository path (chain-configs/testnet/ for an Adiri tag, chain-configs/mainnet/ otherwise), hashed as committed at the tag. It then runs every check from step 6 except the signature count. The last line it prints is SHA256SUMS sha256: <hex>. Write that value down; step 5 asks for it and refuses to sign when the draft’s SHA256SUMS has another hash.

Because release-build rewrites the draft’s notes, add the operator notes after it, above the generated notes, and leave the generated notes as they are: release-publish refuses a body that does not end with them.

gh release view v0.17.0-adiri --json body --jq .body > notes.md
# put the operator notes at the top of notes.md
gh release edit v0.17.0-adiri --notes-file notes.md

5. Sign (laptop)

make release-sign TAG=v0.17.0-adiri

release-sign repeats the tag checks and downloads the draft’s files into target/release-artifacts/v0.17.0-adiri/. It then runs the file checks of release-verify in step 6 before anything is signed: the downloads are the assets GitHub lists, the hashes in SHA256SUMS match, the chain-config ones against the tagged commit, IMAGE_DIGEST matches what the registry reports for :v0.17.0-adiri, the tarball holds the four expected files, its binary is byte-identical to the one in the image pulled by digest, and that binary’s --version names the release. It checks that your key is allowlisted and has not signed this release yet, and that every signature already in SHA256SUMS.asc is a good one by an allowlisted key, and shows the tag, the commit, the image, the content of SHA256SUMS and its SHA-256. It asks for the SHA-256 that release-build printed in step 4 (paste the value or the whole line), or reads it from RELEASE_SUMS_SHA256, and stops without signing when it differs. RELEASE_YES=1 answers only the yes prompt that follows, never this one, so a run without a terminal needs RELEASE_SUMS_SHA256. It signs SHA256SUMS with the key release-tag would use, adds the signature to SHA256SUMS.asc, uploads it, then downloads it again and checks that it is the file it uploaded and that it carries your signature. Sign one at a time: two maintainers uploading at once replace each other’s file, and release-sign stops with an error for the one whose signature was lost. Expect one touch; PIN and touch during signing covers the prompts and retries.

6. Verify and publish (build server)

make release-verify TAG=v0.17.0-adiri
make release-publish TAG=v0.17.0-adiri
docker logout ghcr.io

release-verify checks the whole release:

• every tag check from step 3;
• the draft holds exactly the tarball, IMAGE_DIGEST, SHA256SUMS and SHA256SUMS.asc;
• SHA256SUMS names exactly IMAGE_DIGEST, the tarball and the network’s three chain-config files, and every hash matches, the chain-config ones against the tagged commit;
• enough distinct allowlisted maintainers signed: RELEASE_SIG_THRESHOLD if set, otherwise MIN_SIGNATURES, which is 1;
• every signature in SHA256SUMS.asc is a GOODSIG with a VALIDSIG by an allowlisted key: one that is bad, unverifiable, expired or revoked, or by a key that is not on the allowlist, refuses the release even next to good ones, which is the rule of the operator check on the install page;
• IMAGE_DIGEST matches the registry;
• the tarball holds exactly the four expected files in its directory;
• the binary in the tarball is byte-identical to the one in the image pulled by digest;
• --version in the image shows the version and the tagged commit, and its Build Features line is empty for a mainnet tag and, for an Adiri tag, lists adiri and at most faucet besides, which adiri implies; release-build runs the same check before it pushes the image.

It ends with a line like verified v0.17.0-adiri commit=... signatures=1/1 image=....

release-publish runs the same checks, and also checks the ghcr login when it will move the channel alias, before it asks for yes. While the release is a draft, it checks that the release body ends with the notes generated for the tagged commit and the image digest, line for line, and shows the operator notes above them for you to review. After you type yes, it checks that the draft’s files are still the ones it verified, by name, asset id and digest, and that the body has not changed, and publishes the draft as Latest, as a normal release or as a pre-release, following the channel table. It checks the files and the body once more after publishing, before it moves the alias. If the release is the highest published final version in its channel, it points the channel alias at the image digest and confirms that the registry agrees. It skips work that is already done, so it is safe to run again: when a run stopped after publishing, run make release-publish again, and on the release that is now public it keeps the body as it is, without comparing it with the generated notes, and moves the alias. docker logout ghcr.io removes the registry credential from the build server until the next release.

7. CI verifies the published release

Publishing with your own gh login, as release-publish does, starts the verify-release job, which runs etc/release.sh verify on a GitHub runner with the scripts and the allowlist from main and without registry credentials. The check is detective only: the release is already public when it runs, so a failure means following After publish. GitHub starts no workflow for a publish made with a GITHUB_TOKEN, so a release that someone publishes from a workflow is never checked by this job. After every publish, check that a verify-release run exists for the tag:

gh run list --workflow release.yaml --event release --limit 3

If none of the runs is for v0.17.0-adiri, start one, and use the same command to run the check again later:

gh workflow run release.yaml -f tag=v0.17.0-adiri -f mode=verify

Then follow Installing a release on a clean host, and announce the release with a link to Release notes.

Release candidates

A candidate goes through the same seven steps with an -rcN tag, for example v0.17.0-adiri-rc1. release-prep heads its CHANGELOG.md section with the final version, v0.17.0-adiri, because candidates have no section of their own. A later candidate, or the final release, can tag the same commit: start at step 2, setting RELEASE_COMMIT if main has moved. If fixes have landed since the candidate, start at step 1 with the new tag; release-prep replaces the candidate’s section as long as it is still the newest one. Publishing a candidate never moves an image alias.

What the deployment scripts check

The node deployment scripts in tn-node-deployment install and update nodes from these releases with no person reading the output, so they check every release the same way:

1. They download SHA256SUMS and SHA256SUMS.asc from the release, and the tarball or IMAGE_DIGEST, whichever they install.
2. They build a keyring from the maintainer public keys shipped in their own signed script release, not from this repository, and run gpgv --status-fd 1 --keyring <keyring> SHA256SUMS.asc SHA256SUMS.
3. They require a VALIDSIG line whose primary key fingerprint, the line’s last field, is one of those keys, and refuse the release on any BADSIG, EXPKEYSIG, REVKEYSIG, NO_PUBKEY or ERRSIG line.
4. Before anything is installed, they check each file they use against its line in SHA256SUMS, the network’s committee.yaml, genesis.yaml and parameters.yaml included.

For maintainers this means:

• The release files keep their names and formats: SHA256SUMS in sha256sum format with the chain-config lines under their repository paths, a detached ASCII-armored SHA256SUMS.asc, the tarball and IMAGE_DIGEST. A renamed or reformatted file stops every node update.
• A new key that signs releases, whether a primary key that signs, as every key on the allowlist does today, or a subkey with the sign capability, reaches the scripts in a signed script release before it signs a node release; until then the scripts refuse that release with NO_PUBKEY or ERRSIG. An expired or revoked key stops verifying there as it does here.
• The scripts require one valid signature. Raising MIN_SIGNATURES here does not raise what they require until they change too.

When a step fails

Every step prints error: and the reason when it stops. etc/release.sh exits with 1 when a check fails, 2 for a usage error or an unmet precondition, and 3 when GitHub, the registry, the RPC endpoint or a git or docker network call failed; make shows the code as Error N. GitHub API calls, git fetch and git ls-remote, registry lookups, the pull of the git-cliff image and the attestation check are retried before a step gives up with 3. A request that GitHub refuses, with an HTTP 4xx other than 429, is not retried and exits with 1 or 2, and so does a tag push that origin rejects. git push, docker push, docker pull, cargo update, the image build, the submodule fetch and the alias move are tried once, and exit with 3 when they fail. A signature that gpg cannot make, usually because the YubiKey is not plugged in, exits with 2. After an exit code of 3, run the same command again. If cargo update or the image build fails the same way a second time, the cause is not an outage: the output above it shows it, such as a version conflict or a compile error. If release-build had already pushed the image, the second run stops because :<TAG> is in the registry; rebuild with RELEASE_REBUILD=1 as the table below says. If release-sign had already uploaded the signature, the second run stops because your key has already signed, and nothing more is needed.

Rollback and yanking

Before publish

Until the release is published, the public parts are the tag, the :<TAG> image, and the release commit on main, whose CHANGELOG.md section is already on the docs site’s Release notes page. The tag stays: only maintainers can create a v* tag, and it must be signed; nobody, maintainers included, can update or delete one. Delete the draft, if CI created one, and the image version, if release-build pushed one, running the gh commands on the build server, where the package scopes belong:

gh release delete v0.17.0-adiri --yes
ID=$(GH_TOKEN="$PACKAGES_PAT" gh api --paginate /orgs/Telcoin-Association/packages/container/telcoin-network/versions \
  --jq '.[] | select(.metadata.container.tags | any(. == "v0.17.0-adiri")) | .id')
GH_TOKEN="$PACKAGES_PAT" gh api -X DELETE "/orgs/Telcoin-Association/packages/container/telcoin-network/versions/$ID"

The packages API accepts only a classic personal access token with the read:packages and delete:packages scopes, so put one in PACKAGES_PAT for those two calls and revoke it afterwards; the package’s Versions page on GitHub can delete the version instead.

Then release the next version from step 1, which supersedes the tag: the next release candidate after a withdrawn candidate (v0.17.0-adiri-rc2 after v0.17.0-adiri-rc1, or the final v0.17.0-adiri), and the next patch version after a withdrawn final release (v0.17.1-adiri after v0.17.0-adiri). The version rules refuse a final release or a release candidate whose version is not above every final release in its channel, so a withdrawn final’s version is never released; a withdrawn candidate’s version can still be released, by a later candidate or by its final. Leave CHANGELOG.md as it is. release-prep replaces a withdrawn candidate’s section while it is the newest one; a withdrawn final keeps its section, and the next section lists the commits after its tag. Say in the next release’s operator notes that the version was withdrawn.

After publish

Never delete a published tag, release or image, because operators may have verified and pinned them. Instead:

1. Put a warning at the top of the release notes with gh release edit <TAG> --notes-file notes.md.

2. Mark the release as a pre-release with gh release edit <TAG> --prerelease.

3. If the channel alias points at it, move the alias back to the previous good release from the build server, after make docker-login:

   PREV=vX.Y.Z-adiri
   make release-verify TAG="$PREV" &&
   REF=$(curl -fsSL "https://github.com/Telcoin-Association/telcoin-network/releases/download/$PREV/IMAGE_DIGEST") &&
   docker buildx imagetools create --prefer-index=false --tag ghcr.io/telcoin-association/telcoin-network:adiri "$REF" &&
   docker buildx imagetools inspect ghcr.io/telcoin-association/telcoin-network:adiri --format '{{println .Manifest.Digest}}'
   

   Set PREV to the previous good release. --prefer-index=false copies the image manifest as it is, as release-publish does, so the last command prints the digest after the @ in $REF. For a mainnet release, the alias is :latest.

4. Ship a fixed patch release through all seven steps; publishing it moves the alias forward again.

5. Tell operators the affected version and the version to roll back to, following the release and network update process.

Adding a maintainer or changing the threshold

• The new maintainer follows YubiKey signing setup.
• One PR adds .github/maintainer-gpg-keys/<handle>.asc and the maintainer’s row in the SECURITY.md key table.
• An existing maintainer reviews it and confirms the fingerprint with the new maintainer over a separate channel, such as a call, and against https://github.com/<handle>.gpg.
• The key can sign as soon as the PR merges, because every check reads the allowlist from main.

The threshold is MIN_SIGNATURES in etc/release.sh, currently 1. RELEASE_SIG_THRESHOLD raises it for a single command, and can go neither below MIN_SIGNATURES nor above the number of allowlisted maintainers. To raise it for every release, change MIN_SIGNATURES in a PR that also updates the signatures-required line in SECURITY.md and tells operators on Installing a release how many good signatures to expect. Signatures count per maintainer handle, so several keys in one maintainer’s file count once. With a threshold above 1, each signer runs make release-sign on their own laptop before anyone runs make release-publish.

Rotating or revoking a key

Expiry, loss and rotation has the commands. The allowlist and SECURITY.md change like this:

• for an extended expiry or a spare YubiKey’s key, update <handle>.asc in place, and add a SECURITY.md row for a spare’s key;
• for a replacement YubiKey, which holds a new key, add that key to <handle>.asc with a new SECURITY.md row, and revoke the old key;
• for a revoked key, after a lost YubiKey or a compromise, delete it from <handle>.asc (the whole file when no other key remains), mark its SECURITY.md row revoked YYYY-MM-DD, and remove it from GitHub as Lost or broken YubiKey describes;
• never add a placeholder file for a new maintainer; the file is added when the key exists.

release-verify does not count a signature from an expired or revoked key, and it reads the keys from main. So once a key expires or is revoked, CI verification of the older releases it signed fails. Extend keys before they expire. After revoking a compromised key, tell operators which releases are no longer trusted.