Rotate a release signing key without locking users out

Corrected, November 2026. An earlier version of this part told you to publish the new key in the WKD bucket your installed copies already read. Do that and they refuse every further update, because a tool on the default both key source needs its embedded keys and its WKD keys to match exactly. It also said the rotation authority could bridge you to a new key, and it can’t. Both were my mistakes, and the steps below are the ones that work. If you’ve already published that way, put that bucket back to what your shipped copies embed, and they’ll verify again as long as you’re still signing with the old key. Anything you built with the bigger set against that address will refuse until it’s reinstalled.

Most signing guides stop the moment the first release goes out the door, which is a shame, because the question that keeps you up at night comes later: what do you do when the key has to change? Keys get rotated on a schedule, keys get compromised, and one day you’ll want to move off RSA onto something newer. This last part of the signing series covers the bit everyone skips, swapping a signing key out from under a fleet of installed tools without locking them out, and what you’re left with on the day the old key is already gone.

By Part 6 you’ve a pipeline signing tagged releases through a KMS key, a public key published over WKD, and that same key baked into the binary as its trust anchor. It all works. The catch nobody mentions is that you’ve now got a key embedded in software sitting on other people’s machines, and you can’t quietly change that copy. So let’s plan for changing it properly.

This is the walkthrough from the go-tool-base docs, which is the copy I keep up to date: https://gtb.phpboyscout.uk/tutorials/rotate-your-signing-key/. The steps are checked there with local keys standing in for KMS. You’ll need gtb v0.50.4 or later, the first release that gives a generated tool the authority/ directory used below. Allow an afternoon for the ceremony and the first release, then weeks of overlap before you retire the old key.

Why there’s no auto-rotate button

If you’ve used KMS for encryption, you’ll know it can rotate keys for you on a yearly tick. Asymmetric SIGN_VERIFY keys don’t get that, and the terraform-aws-signing-kms module sets enable_key_rotation = false on purpose. A new key means a new public half, and copies of the old public half are compiled into binaries you’ve already shipped, so this was never something a cloud provider could do behind your back.

So rotation here is a runbook you maintain, not a checkbox you tick. In exchange you’ve got a key with no exfiltration path, and for a release-signing key I’d want to be standing right there when it changes anyway.

The two rules that shape the procedure

Both of these come from how your tool checks an update:

An exact match. The embedded keys and the WKD keys have to agree. The tool fetches the WKD keys for the email compiled into it and refuses any update if their fingerprints differ from the embedded set. That refusal is always fatal.
A fixed set. A shipped binary’s embedded keys never change, so the WKD bucket it reads can’t change either.

Put those together and a WKD bucket is frozen from the moment you publish it. Adding v2 to it would make your installed copies refuse every update. So each generation of your tool gets its own WKD identity. Old binaries keep reading the frozen [email protected] bucket, and new binaries read a new [email protected] bucket holding the larger set.

The plan

Re-mint v1 under the new identity, keeping its fingerprint.
Mint v2.
Give the rotation authority the new identity.
Publish the new bucket, and leave the old one alone.
Ship a release that trusts the new set.
Sign with both keys through an overlap.
Prove both generations verify.
Close the window.

Before you start

The v2 KMS key. A second instance of the module, named acme-release-signing-v2, applied as in Part 2. Don’t edit the v1 module: its name and key_spec are fixed, and v1 has to keep signing until the end.
An inventory of the keys you’ve embedded: fingerprint, user ID email and creation time for each. For a key file:

gpg --show-keys --with-colons signing-key-v1.asc | \
  awk -F: '/^pub/{print "created:", strftime("%Y-%m-%dT%H:%M:%SZ", $6, 1)}
           /^fpr/{print "fpr:", $10} /^uid/{print "uid:", $10}'

A copy of your published WKD tree that’s byte-identical to what’s served. Check each file’s sha256sum against that file fetched with curl.
The rotation authority’s private half, out of offline storage, for one step.

Re-mint v1 under the new identity

A key only appears in a WKD bucket if one of its user IDs names that bucket’s email. So v1 has to gain the [email protected] identity without becoming a different key. Mint it again with the new email and the original creation time from your inventory:

gtb keys mint \
    --backend aws-kms \
    --kms-region eu-west-2 \
    --key-id alias/acme-release-signing-v1 \
    --name "Acme Releases" \
    --email [email protected] \
    --created "2026-06-02T00:00:00Z" \
    --output signing-key-v1.asc

The fingerprint it logs must match v1’s fingerprint from the inventory. If it doesn’t, the --created value is wrong, so fix that before going any further. Nothing has been published yet, so a mistake here costs nothing.

Mint v2

The new key gets the new identity and no --created pin, because it’s new:

gtb keys mint \
    --backend aws-kms \
    --kms-region eu-west-2 \
    --key-id alias/acme-release-signing-v2 \
    --name "Acme Releases" \
    --email [email protected] \
    --output signing-key-v2.asc

Record the creation time it logs beside the alias. It’s part of v2’s fingerprint for good.

Give the rotation authority the new identity

The rotation authority needs the new user ID too, keeping its fingerprint. This is the one step that touches its private half, so do it on a trusted machine, in a throwaway keyring:

export GNUPGHOME="$(mktemp -d)"
gpg --import rotation-authority.priv.asc
gpg --quick-add-uid <rotation-authority-fingerprint> \
    "Acme Rotation Authority <[email protected]>"
gpg --armor --export <rotation-authority-fingerprint> > rotation-authority.asc
gpg --show-keys rotation-authority.asc
rm -rf "$GNUPGHOME" && unset GNUPGHOME

gpg --show-keys should list both user IDs under one fingerprint:

      3023D464AF06360994F6E4A555E8928F70BD4408
uid                      Acme Rotation Authority <[email protected]>
uid                      Acme Rotation Authority <[email protected]>

Then put the private half back in storage. Only the public export leaves the ceremony.

Publish the new bucket, and only the new bucket

Now we build a tree for the new identity alone. Passing only the new email means the frozen bucket can’t be regenerated by accident:

gtb keys wkd \
    --domain acme.dev \
    --email [email protected] \
    --output ./wkd-new \
    signing-key-v1.asc signing-key-v2.asc rotation-authority.asc

INFO  WKD bucket [email protected] hash=yxidni38ndxos3irk9hcm9bjnnemdx8r keys=3
INFO  WKD tree complete output=./wkd-new method=advanced emails=1 files=2

Check the bucket holds the three keys your next release will embed:

gpg --show-keys wkd-new/.well-known/openpgpkey/acme.dev/hu/yxidni38ndxos3irk9hcm9bjnnemdx8r

Copy only the new hu/ file into your staging copy of the live tree, refusing to overwrite anything already there:

cp --update=none wkd-new/.well-known/openpgpkey/acme.dev/hu/yxidni38ndxos3irk9hcm9bjnnemdx8r \
   wkd-staging/.well-known/openpgpkey/acme.dev/hu/

Check the existing files are still byte-identical to production, then deploy the way you did before. Afterwards, compare each served file’s sha256sum with the staged one. The old bucket matching is your proof that the installed copies are untouched.

Ship a release that trusts the new set

In your tool, point it at the new bucket first. On an older scaffold, this is also the step that creates internal/trustkeys/authority/ and moves the tool onto the gtb release you’re running. Then replace the embedded keys with the new-identity set:

gtb enable signing --email [email protected]
cp signing-key-v1.asc signing-key-v2.asc internal/trustkeys/keys/
cp rotation-authority.asc internal/trustkeys/authority/

The re-minted v1 replaces the old file. It’s the same key, so the fingerprint set is v1, v2 and the rotation authority, which is what the new bucket serves.

If you followed an earlier version of Part 5, clear the old copy of the authority first. That version put rotation-authority.asc in keys/, and it must not stay there once it’s in authority/ as well:

git rm internal/trustkeys/keys/rotation-authority.asc

Your tool loads both directories, so a key left in both counts twice. The set compiled into that release then can’t match the bucket, and copies of it refuse all later updates. Check before you build that the file is in authority/ and nowhere else.

Don’t switch signing to v2 yet. Copies already installed only trust v1, and they need to be able to verify the release that brings them v2.

Sign with both keys during the overlap

For the overlap window, each release carries two signatures in one checksums.txt.sig: one from v1 for the installed copies, one from v2 for the new ones. sigillum sign --append adds a second signature packet to an existing signature file, and verifiers skip a packet whose key they don’t have.

The generated signs block calls sigillum sign once and can’t name a second key yet (#156). For the window, replace it with a script, scripts/sign-release.sh:

#!/bin/sh
# Usage: sign-release.sh <signature> <artifact>
set -eu
sig=$1 artifact=$2

sigillum --ci sign --backend "$SIGN_BACKEND" --key-id "$SIGN_KEY_ID" \
  --public-key "$SIGN_PUBLIC_KEY" --output "$sig" "$artifact"

if [ -n "${SIGN_KEY_ID_2:-}" ]; then
  AWS_ROLE_ARN="${AWS_ROLE_ARN_2:-${AWS_ROLE_ARN:-}}" \
    sigillum --ci sign --append --backend "$SIGN_BACKEND" --key-id "$SIGN_KEY_ID_2" \
    --public-key "$SIGN_PUBLIC_KEY_2" --output "$sig" "$artifact"
fi

Make it executable, point the signs block at it, and tell gtb to leave the release config alone while it’s hand-edited:

signs:
  - id: checksums
    cmd: scripts/sign-release.sh
    env:
      - "AWS_ACCESS_KEY_ID="
      - "AWS_SECRET_ACCESS_KEY="
    args: ["${signature}", "${artifact}"]
    artifacts: checksum
    signature: "${artifact}.sig"
    output: true

Keep the env: lines on GitLab, where the generated block has them too. The goreleaser component hands the job the release store’s keys, and static keys outrank the OIDC token, so both signatures would otherwise fail. On GitHub leave them out, because the role step supplies its credentials through those variables.

chmod +x scripts/sign-release.sh
gtb ignore add .goreleaser.yaml

In the release job, set the first key to v1 and the second to v2:

  variables:
    SIGN_BACKEND: aws-kms
    SIGN_KEY_ID: alias/acme-release-signing-v1
    SIGN_PUBLIC_KEY: internal/trustkeys/keys/signing-key-v1.asc
    SIGN_KEY_ID_2: alias/acme-release-signing-v2
    SIGN_PUBLIC_KEY_2: internal/trustkeys/keys/signing-key-v2.asc
    AWS_ROLE_ARN: ${SIGNER_ROLE_ARN}
    AWS_ROLE_ARN_2: ${SIGNER_ROLE_ARN_V2}

One OIDC token works for both roles, because both trust your project’s tag pipelines. sigillum sign reads the key’s region from --kms-region, which defaults to eu-west-2, so add it to both commands in the script if yours differs.

Prove both generations verify

Before you rely on the window, check a released checksums.txt.sig against each generation. The frozen bucket is, byte for byte, the key set your installed copies hold, so use it as the old keyring:

gpg --list-packets checksums.txt.sig | grep -c 'signature packet'

export GNUPGHOME="$(mktemp -d)"
curl -s https://openpgpkey.acme.dev/.well-known/openpgpkey/acme.dev/hu/y84sdmnksfqswe7fxf5mzjg53tbdz8f5 | gpg -q --import
gpg --verify checksums.txt.sig checksums.txt
rm -rf "$GNUPGHOME"

export GNUPGHOME="$(mktemp -d)"
gpg -q --import signing-key-v2.asc
gpg --verify checksums.txt.sig checksums.txt
rm -rf "$GNUPGHOME" && unset GNUPGHOME

Each check should give one good signature and one it can’t check:

gpg: Good signature from "Acme Releases <[email protected]>" [unknown]
gpg: Can't check signature: No public key

The No public key line is the other generation’s packet being skipped. It’s correct, not an error.

Close the window

Leave the overlap running until you’re confident the slow movers have updated: weeks, not hours, depending on how often your users run the tool. Then:

Stop signing with v1. Delete the _2 variables and set the first key to v2. The script’s second pass goes quiet.
Hand the release config back to gtb, if you like: gtb ignore remove .goreleaser.yaml, then gtb enable signing --key-id alias/acme-release-signing-v2 --public-key internal/trustkeys/keys/signing-key-v2.asc.
Only now retire the v1 KMS key. Destroying it any earlier strands the copies that haven’t updated.

Keep v1’s public half embedded, and keep the frozen bucket published, for as long as anything v1 signed should verify. For released software that’s indefinitely.

Expect your own tests to complain during the rotation. A test that pins your embedded fingerprints should fail when v2 arrives, and updating it in that change is the deliberate acknowledgement it exists to force.

When the old key is gone

Everything above assumes v1 can still sign through the overlap. If it’s gone (the account closed early, access lost, or you daren’t use it again), there’s no overlap to run. Mint a fresh release key and follow the same procedure from the top: a new identity and a new WKD bucket, the new key set embedded, and releases signed with the new key alone.

Copies that only trust v1 can’t verify those releases, so they’ll refuse to update. Tell your users to reinstall once from the release page. The new binary trusts the new key, and self-update works again from there. That’s the cost of losing a key early, and it’s why the overlap matters.

Don’t sign a release with the rotation authority to bridge the gap. Your tool refuses a release signed by it. The plan is for it to authorise a new key set in an emergency, but that path isn’t built yet: it’s a draft spec, and nothing in go-tool-base or go/signing v0.11.2 reads the document it would sign. So for now the reinstall is what you’ve got.

Keep the private half safe regardless, as generate the rotation-authority key describes: a paperkey printout and an encrypted USB stick, kept apart, and the backup typed back in once to prove it reads. A break-glass key you can’t restore is just a pane of glass.

The whole chain

That’s the series. The key is born in KMS and never leaves it (Part 2). Its public half is minted out of the HSM, published on WKD somewhere your release host can’t touch, and embedded in your binary (Part 4 and Part 5). Tagged releases are signed through KMS over short-lived OIDC credentials (Parts 3 and 6). Installed copies check their own updates against two independent copies of the key (the cross-check). And when the key has to change while you’ve still got the old one, it changes without stranding anyone.

If you’ve dipped in for one piece, the pillar ties the lot together. From here, the docs are the place to go deeper:

Release-binary signing: the whole design in one place.
Sign release artefacts: --append and the other sign flags.

Sign your own binaries with go-tool-base, part 7: rotation and break-glass

Rotate a release signing key without locking anyone out, and what you're left with if the old key is already gone. Why asymmetric KMS keys have no auto-rotate button.