Orienteering on PHP Boy Scout

Sign your own binaries with go-tool-base, part 7: rotation and break-glass

Tue, 23 Jun 2026 00:00:00 +0000

Most signing guides stop the moment the first release goes out the door, which is a shame, because the question that keeps you up at night comes later: what do you do when the key has to change? Keys get rotated on a schedule, keys get compromised, and one day you’ll want to move off RSA onto something newer. This last part of the signing series covers the bit everyone skips, swapping a signing key out from under a fleet of installed tools without locking a single one of them out, and the break-glass key for the day the primary is gone.

By Part 6 you’ve a pipeline signing every tagged release through a KMS key, a public key published over WKD, and that same key baked into the binary as its trust anchor. It all works. The catch nobody mentions is that you’ve now got a key embedded in software sitting on other people’s machines, and that key is the one thing in the whole chain you can’t quietly change. So let’s plan for changing it properly.

Why there’s no auto-rotate button

If you’ve used KMS for encryption, you’ll know it can rotate keys for you on a yearly tick. Asymmetric SIGN_VERIFY keys don’t get that, and the terraform-aws-signing-kms module sets enable_key_rotation = false on purpose. That’s not an oversight to work around; it falls straight out of how the key is built. The private half of a signing key never leaves the HSM and has no export path at all, so there’s no mechanism by which AWS could hand your verifiers a rotated public half and keep the old one verifiable. KMS auto-rotation works for symmetric keys precisely because you never see the key material; the instant you need a public key your clients pin against, rotation stops being something a cloud provider can do behind your back.

So rotation here is a runbook you maintain, not a checkbox you tick. That sounds like the worse deal until you weigh it: you’ve got a key with no exfiltration path, in exchange for rotating it by hand on the rare occasions you must. For a release-signing key that’s exactly the right trade. You don’t want this key rotating itself; you want to be standing right there when it happens.

Rotate by minting a new key, never by changing the old one

Here’s the rule that makes everything else fall into place. You do not edit the existing key. The module’s key_spec and name are immutable by design, and that immutability is a feature: the v1 key is a fixed point that stays verifiable while you stand up its replacement next to it.

So you mint a second key. A second module instance, named acme-release-signing-v2, with its own alias alias/acme-release-signing-v2:

module "signing_kms_v2" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v2"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn

 ci_subject_filters = [
 "project_path:acme/acme-cli:ref_type:tag:ref:v*",
 ]

 key_administrator_arns = [/* ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

Apply that and the v1 key carries on signing, undisturbed. Now mint its public half exactly the way you minted v1 back in Part 4, only the alias changes:

gtb keys mint \
 --backend aws-kms \
 --kms-region eu-west-2 \
 --key-id alias/acme-release-signing-v2 \
 --name "Acme Releases" \
 --email release@acme.dev \
 --created "2026-06-08T00:00:00Z" \
 --output signing-key-v2.asc

You’ve now got two real signing keys, both live, neither one a threat to the other. The whole rotation is the careful business of moving traffic from one to the other while every installed client keeps verifying.

The dual-publish, dual-sign window

This is the part that does the work, and it leans on one fact about how the verifier treats your keys. The trust anchor isn’t a single key, it’s a set: v1, v2, and the rotation authority all sit in internal/trustkeys/keys/ and all get embedded together. A release verifies if any key in that set validates its signature. That’s what makes a handover window possible, because during it a client might hold v1, or v2, or both, and verify happily whichever it has.

So you run an overlap. Three moves, in order.

First, publish both public keys over WKD. Same email bucket, same command you already know from Part 4, just with v2 added to the file list:

gtb keys wkd \
 --domain acme.dev \
 --email release@acme.dev \
 --output ./wkd-staging \
 signing-key-v1.asc signing-key-v2.asc rotation-authority.asc

Both signing keys share release@acme.dev in their UID, so they land in one hu/ bucket together, and the rotation authority rides along as it always has. Deploy that staging tree the way you deployed it before. Now the WKD endpoint serves the new trust anchor, so a client that fetches keys on its next update picks v2 up automatically.

Second, embed v2 alongside v1 and ship a release. Drop signing-key-v2.asc into internal/trustkeys/keys/ next to v1, cut a release, and that build now ships knowing about both keys. Installed clients pick the new trust anchor up as they update through the window. This is the slow bit, and it should be: you want the new key spread far and wide before you stop signing with the old one.

Third, sign with v2, wait, then retire v1. Point the pipeline’s GTB_SIGNING_KEY_ID at alias/acme-release-signing-v2 so new releases are signed by the new key. A client that’s already updated verifies against the embedded v2; one that’s lagging still has v1 in its set and, because you’re still publishing v1 over WKD, can still fetch and trust it. Nobody’s locked out.

Leave that overlap running long enough that you’re confident the slow movers have updated, weeks, not hours, depending on how often your users actually run the thing. Only then do you retire v1: drop its module instance, pull signing-key-v1.asc from the embed directory and the WKD file set, and ship a release that knows only about v2. The rotation is done, and at no point did an installed tool see a signature it couldn’t check.

The break-glass key, for when there’s no handover

Everything above assumes the old key can hand over to the new one: it’s still there, still able to sign, and you’re rotating on your own terms. The nightmare is the other case. The KMS key is gone, access revoked, or you’ve reason to think it’s compromised and you daren’t sign anything with it again. There’s no handover, because the thing that would do the handing is exactly what you’ve lost.

That’s what the rotation-authority key is for, and it’s why Part 4 had you mint it the moment everything else was calm. It’s an offline ed25519 key, generated once on a trusted machine, and the go-tool-base how-to, generate the rotation-authority key, walks the storage in detail. The short version: the private half never lives on a networked box. You print a paper backup with paperkey, write it to an encrypted USB stick, and the two go in a safe; the local copy gets shredded. The how-to even has you type the paper backup back in once before you walk away, because discovering your printer ate a stripe of pixels is a problem you want now, not eighteen months from now when the building’s on fire.

The public half, though, has been in your trust set the whole time, embedded in the binary and served over WKD right alongside the signing keys. That’s the trick. Because every installed client already trusts the rotation authority, it can vouch for a brand-new signing key outside the normal sign-with-the-old-key path. You bring the private half out of the safe, use it to authorise the new key, ship that, and installed tools adopt the replacement on their next update, all without the dead primary key ever having to sign a thing.

It is, deliberately, a key you hope never to touch. But a break-glass key you forgot to cut is just a pane of glass.

The whole chain, end to end

That’s the series. Step back and the shape is one clean line: a signing key is born inside KMS and never leaves it (Part 2); its public half is minted out of the HSM without the private bytes ever surfacing (Part 4); that public key is published off-platform over WKD, somewhere your release host can’t quietly rewrite (Part 4), and embedded into the binary as a required trust anchor (Part 5); every tagged release is signed through the key over short-lived OIDC credentials with no stored secrets (Parts 3 and 6); a stranger’s copy of your tool verifies its own updates against that anchor before trusting a byte (the cross-check); and when the day comes, the whole thing is rotatable without locking anyone out. A key that can’t be stolen, can’t be forged, and can still be replaced.

If you’ve followed all seven parts, you’ve built that. If you’ve dipped in for one piece, the pillar ties the lot together and points at the deep-dives behind the why. Either way, your users are getting updates they can actually trust, which was the whole point. Go and leave your supply chain better than you found it.

Sign your own binaries with go-tool-base, part 6: sign every release with GoReleaser

Mon, 22 Jun 2026 00:00:00 +0000

By now you’ve got all the pieces lying on the bench. A KMS key that signs but never hands over its private half (Part 2). A CI role you can assume over OIDC with no stored credentials (Part 3). A public key you’ve minted and published (Part 4), and embedded in the binary (Part 5). What you don’t have yet is the bit that makes it routine: a release that signs itself, every time, without you remembering to do anything. That’s this part. We wire signing into the tagged-release pipeline so that pushing a v* tag is the whole ceremony.

This is the part of the signing series where the chain stops being a set of commands you run by hand and becomes something the pipeline does for you. We’re using GoReleaser, which already builds your binaries, writes a checksums file and cuts the release. It needs one extra trick: sign the checksums on the way out, through the KMS key, using credentials that only exist for the length of the job. And you don’t hand-write that trick into your release config. gtb does.

You’ll need a working GoReleaser setup releasing a go-tool-base CLI, plus the KMS key, the signer role and the embedded public key from the earlier parts.

One command wires the signing in

In Part 5 you ran gtb enable signing to turn on the verifying side: embed the key, check every update against it. Now you give that same command the key the release pipeline should sign with, and it wires the producing side too:

gtb enable signing --key-id alias/acme-release-signing-v1

That records the KMS key in your .gtb/manifest.yaml and regenerates .goreleaser.yaml with a signs: block that calls gtb sign. The WKD email and everything else you set in Part 5 stay exactly as they were; you’re adding the key, not starting over. Because the block is generated, you don’t hand-edit the release config any more than you hand-edit the embed wiring. Change the key later (a new region, a rotated alias) and you re-run the command, not the YAML.

The generated block looks like this:

signs:
 - id: checksums
 cmd: gtb
 args:
 - "--ci"
 - "sign"
 - "--backend"
 - "aws-kms"
 - "--kms-region"
 - "eu-west-2"
 - "--key-id"
 - "alias/acme-release-signing-v1"
 - "--public-key"
 - "internal/trustkeys/keys/signing-key-v1.asc"
 - "--output"
 - "${signature}"
 - "${artifact}"
 artifacts: checksum
 signature: "${artifact}.sig"
 output: true

The region defaults to eu-west-2 and the public key to the internal/trustkeys/keys/signing-key-v1.asc you embedded in Part 5; pass --kms-region or --public-key if yours differ. The backend defaults to aws-kms, which is the one that matters in CI.

Why only the checksums

artifacts: checksum is the line doing the load-bearing work. It tells GoReleaser to run the signing command once, over the checksums manifest only, not over every binary and archive in the release.

Here’s why that’s enough. GoReleaser builds your binaries, then writes a checksums.txt listing the SHA-256 of each one. Every artefact in the release is named in that file by its hash. So if you sign the manifest, you’ve transitively vouched for everything it lists: change a single byte of any binary and its hash no longer matches the line in checksums.txt, and the moment you alter checksums.txt to cover for that, the signature over it breaks. One signature, the entire release covered, through the hash chain. The per-binary build stays completely untouched, which keeps reproducible builds reproducible. signature: "${artifact}.sig" names the output checksums.txt.sig, the detached, ASCII-armored OpenPGP signature your tool looks for on every self-update.

No shim, just `gtb`

If you go reading go-tool-base’s own release config, you’ll find its signs: block points at a scripts/sign-release.sh shim rather than calling gtb directly. Yours doesn’t, and the difference is worth understanding.

go-tool-base is signing the very binary it’s in the middle of building, so it can’t use an installed gtb to do it. Its shim runs go run ./cmd/gtb to build a throwaway signer from source. It also reads the key id, public key and region from environment variables, because the one config has to serve every build. Your tool has neither problem. It calls the gtb you already installed, and the generator already knows your key id, region and public-key path because you just told it. So there’s nothing for a shim to abstract: the whole invocation goes straight into args:, where you can read it.

The one thing you do still need is gtb on the release runner’s PATH. Install it in a before_script, or bake it into your CI image, the same way you would any other release tool.

Where the credentials come from

gtb sign asks KMS to sign. KMS will only oblige if the caller has credentials, and the whole point of keyless CI is that there are no stored credentials to leak. So they’re minted on the fly.

On GitLab, the release job declares an id_tokens: block. GitLab injects a short-lived OIDC token (a JWT) into the job, the before_script writes it to a file, and the AWS SDK’s default credential chain picks it up from there. No aws CLI call, no assume-role-with-web-identity you write yourself: set AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE and the SDK does the web-identity exchange the first time gtb sign touches KMS:

goreleaser:
 id_tokens:
 AWS_WEB_IDENTITY_TOKEN:
 aud: sts.amazonaws.com
 variables:
 AWS_REGION: eu-west-2
 AWS_ROLE_ARN: arn:aws:iam::…:role/acme-release-signing-v1-signer
 AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/oidc-token
 before_script:
 - echo "$AWS_WEB_IDENTITY_TOKEN" > "$AWS_WEB_IDENTITY_TOKEN_FILE"

The key id and region live in the generated signs: block now, not here, so CI only supplies the AWS credentials. AWS_REGION still earns its place: the --kms-region flag points the KMS client at the key, while the SDK uses AWS_REGION for the STS exchange that mints the credentials in the first place. The aud has to match the audience the signer role’s trust policy expects (Part 3 set this up; for the OIDC provider go-tool-base uses, that’s sts.amazonaws.com). On GitHub the moving parts are the same, you just let aws-actions/configure-aws-credentials do the token-to-credentials dance instead of writing the file yourself.

Those credentials are scoped tight. The signer role’s trust policy pins it to this project’s tag pipelines, so even if the role ARN leaked, nothing but a release tag on your repo can assume it.

Don’t sign when there’s nothing to sign with

A local goreleaser release --snapshot, or a CI run that isn’t a release, has no OIDC token and no business reaching for KMS. GoReleaser is told to skip the whole signing step in that case: the release job runs with --skip=sign unless the web-identity token is present. So a non-release build never so much as looks at KMS, which is exactly what you want when you’re iterating on your laptop. The signing only fires on the real thing.

Cut a release

With all of that in place, releasing is one push:

git tag v1.4.0
git push origin v1.4.0

The tag pipeline fires. GoReleaser builds the binaries, writes checksums.txt, calls gtb sign, gtb asks KMS to sign over OIDC, and checksums.txt.sig lands next to the manifest. Both get attached to the release. Nobody typed a signing command and no private key was anywhere near the runner.

A two-person gate, if you want one

There’s a window worth thinking about: a compromised CI runner during a release could, in principle, ride the OIDC credentials to get one malicious thing signed. You can shut that window with an approval gate in front of the signing job. On GitLab, a protected environment with a required approval (or a manual job) makes the release wait for a second pair of eyes; on GitHub, “required reviewers” on the release environment does the same. It’s optional, and it adds friction to every release, so weigh it against how exposed your runners are. For a lot of projects the OIDC scoping alone is enough; for anything where a forged release would be a genuine incident, the gate is cheap insurance.

Where this leaves you

Every release you cut from here on carries a signature made by a key you control and verifiable by anyone, and you got there by pushing a tag. That closes the loop the series has been building toward: the production side now actually produces signed releases.

Which means you can finally pull the trigger on the bit Part 5 left primed. We embedded the public key and left enforcement off, because turning it on before you ship signatures would brick every update. Now that signatures are shipping, Part 7 deals with the part everyone skips and nobody can afford to: rotating the key, and getting yourself out of trouble if one ever goes bad, without locking your users out.

Sign your own binaries with go-tool-base, part 5: embed the key and require verification

Sun, 21 Jun 2026 00:00:00 +0000

By now you’ve got a public key your tool can publish off-platform: minted from a KMS-held private key in Part 4 and served over WKD. That’s half the trust loop. The other half lives inside the binary itself: the tool has to hold a copy of the key it expects so that when an update lands, it can check the signature against something an attacker who owns the release page can’t quietly swap. This part bakes that trust anchor in, wires the self-updater to use it, and turns enforcement on without locking out the people who already have your tool installed.

That last clause is the one that bites, so we’ll come to it slowly. First, turning signing on.

Enable signing with one command

Your root command is generated by gtb, so you don’t wire signing in by hand-editing it. You turn the feature on and let the generator do the wiring:

gtb enable signing --email release@acme.dev

That one command does three things, all in generated, regenerable code you never touch:

scaffolds an internal/trustkeys package that //go:embeds your release keys and hands them to the self-updater;
wires Signing: props.SigningConfig{EmbeddedKeys: trustkeys.Keys()} into your generated root command;
writes a signing.go holding the enforcement defaults, generated from a signing block it adds to your .gtb/manifest.yaml. You change posture by re-running the command, never by editing the file.

Signing is off until you run this, on purpose: it needs a key and a published WKD endpoint, so a freshly generated tool doesn’t carry it uninvited. (If you’re curious what the embed package looks like, it’s a small //go:embed all:keys over an internal/trustkeys/keys/ directory with a Keys() [][]byte accessor. The all: prefix is load-bearing: a plain //go:embed keys won’t compile over a directory that holds only a dotfile, so the scaffold keeps a .gitkeep there. You don’t write any of it.)

Drop your key in

The scaffold gives you an empty internal/trustkeys/keys/. Put the public key you minted in Part 4 into it, alongside the break-glass key:

cp signing-key-v1.asc internal/trustkeys/keys/
cp rotation-authority.asc internal/trustkeys/keys/

The -v1 in the filename isn’t decoration: you’ll rotate one day and embed signing-key-v2.asc alongside it for a release or two (Part 7). The rotation-authority key rides along the same way. Rebuild, and trustkeys.Keys() now returns them. With no .asc present it returns nothing and verification stays dormant, so enabling signing before you have a key breaks nothing.

The WKD cross-check, via `--email`

The --email you passed is doing real work. The embedded key alone is a static anchor (how the verification works): it only ever says “this was the key on the day I was built.” Pairing it with the live WKD copy you published in Part 4 gives a second, independent source the release platform can’t reach. The default key source is both, an embedded-plus-WKD CompositeResolver, and the email is what lets the updater derive the WKD URL. Leave --email off and both quietly degrades to embedded-only.

For a locked-down tool that should refuse to update when it can’t reach WKD, rather than fall back to the embedded key, enable the strict cross-check:

gtb enable signing --email release@acme.dev --require-external-crosscheck

Most tools want the softer default, where WKD strengthens verification when it’s reachable and the embedded key still works when it isn’t.

Confirm the cross-check is actually firing

It’s easy to get this wrong by leaving the email out, and when you do, nothing complains: the updater quietly verifies against the embedded key alone and carries on. The way to know which anchors were actually consulted is to read the log. Every update prints a line naming the resolver it used:

INFO signature verified resolver=composite[embedded,wkd:openpgpkey.acme.dev]

That resolver= field is the whole tell:

composite[embedded,wkd:...] is what you want: both the embedded key and the WKD-served key were fetched, their fingerprints agreed, and the signature checked against the result. The cross-check is live.
embedded means only the baked-in key was used and WKD was never consulted. That’s the silent-degrade trap: key_source is "both", but with no external email there was no WKD URL to derive, so it fell back to a single anchor. If you see this after passing --email, the value didn’t take.
wkd:... on its own is the reverse: WKD was consulted but nothing was embedded.

There’s a matching update signature verification configured resolver=... line at the very start of an update, before any network call, if you’d rather see the choice before the fetch. Two failure shapes are worth recognising too. WARN composite resolver failed (RequireAll=false, continuing) means the WKD fetch fell over (a 404, a flaky network) and the update carried on against the embedded key alone, the soft default you can harden with --require-external-crosscheck. ERROR ErrKeyResolverMismatch is the one you want to see fire in anger: the embedded and WKD keys disagreed, which is exactly the tamper alarm the whole scheme exists to raise (the same mismatch the verification deep-dive walks through).

Require it, in the right order

Here’s the part everyone trips over. Enabling signing does not yet require it: require_signature stays off, and that’s deliberate. Turning it on too early breaks self-update for everyone who already has your tool.

Think about what an existing install holds. A user on an old version has a binary built before you enabled signing. It has no trust anchor. If the first thing it ever sees is a signed, signature-required release, it has nothing to check the signature against, and the update is refused. You’ve locked out exactly the people you were protecting.

The fix is to ship the key ahead of the requirement, so the anchor is already on the user’s machine by the time the first mandatory signature arrives. gtb follows the rollout in its own docs/development/phase2-signing-prep.md, across three releases:

Ship the key before you require the signature, or you lock out every install that predates it. In order:

Release N+1. Enable signing (above) and ship it, with require_signature off. Existing installs pull this update on checksum alone and pick up the embedded key as a side effect.

Release N+2. Ship your first signed release (Part 6 wires GoReleaser). Still not required: the signature is verified when present but not enforced, so nothing breaks.

Release N+3. Now, and only now, turn enforcement on.

When you reach N+3, it’s one command, not a code edit:

gtb enable signing --require-signature

That flips require_signature in the manifest and regenerates signing.go, so the change is tracked and reproducible. Skipping the middle release, requiring a signature before you’ve shipped one, is the exact mistake the ordering exists to prevent; gtb’s own rollout cites v0.12.2 as its first signed release for precisely this reason.

The checksum floor from Phase 1 sits underneath all this and is already on; signature verification adds to it, it doesn’t replace it. And even with signatures required, an end user genuinely stuck on a legacy release can escape with the update.require_signature config key or your tool’s <PREFIX>_UPDATE_REQUIRE_SIGNATURE=false. It’s an escape hatch, not the front door, but it means a requirement you turn on can’t permanently strand anyone.

Where this leaves you

Your binary now carries the key it expects, checks every update against that key and its live WKD twin, and refuses anything that doesn’t match, all without a gpg install on the user’s side and without stranding the installs that came before the key. The trust loop you built by hand in Part 1 now runs on its own, inside a stranger’s copy of your tool.

The one thing still missing is the signature itself on each release. Right now nothing is actually producing checksums.txt.sig. Part 6 wires the KMS signing from Parts 2 and 3 into a real GoReleaser pipeline, so every tagged release comes out signed without you touching a key.

Sign your own binaries with go-tool-base, part 4: mint and publish your public key

Sat, 20 Jun 2026 00:00:00 +0000

By the end of Part 3 your release pipeline can sign through a KMS key it never holds, over credentials that expire in minutes. The private half is locked away exactly where you want it. There’s a snag, though: a signature is no use to anyone who can’t get hold of the matching public key, and a KMS key won’t hand you one. KMS deals in raw signing operations, not OpenPGP entities. So this part does two things: it produces the published public key from the KMS key without ever touching the private bytes, then puts that key somewhere your release platform can’t reach.

That last bit is the part people skip, and it’s the part that does the real work. The whole scheme in a signature the platform can’t forge rests on the verifying key living somewhere an attacker can’t poison in the same breath as the release. We’ll come back to why that matters once the key is in hand.

Mint the public key from KMS

gtb keys mint builds an OpenPGP public key out of a signing backend. In Part 1 the backend was a .pem on disk; now it’s aws-kms, and only that flag changes:

gtb keys mint \
 --backend aws-kms \
 --kms-region eu-west-2 \
 --key-id alias/acme-release-signing-v1 \
 --name "Acme Releases" \
 --email release@acme.dev \
 --created "2026-06-02T00:00:00Z" \
 --output signing-key-v1.asc

INFO Minted OpenPGP key backend=aws-kms key_id=alias/acme-release-signing-v1 output=signing-key-v1.asc creation_time=2026-06-02T00:00:00Z fingerprint=...

Here’s the neat part, and it’s worth pausing on. An OpenPGP key isn’t just a lump of public-key material; the key carries a self-signature, a signature it makes over itself to bind the name and email to the key. So minting a public key normally needs the private key to do that signing. mint doesn’t. It wraps the backend in a crypto.Signer, and every signing operation, the self-signature included, becomes a kms:Sign call. KMS does the maths inside the HSM and hands back a signature; no private byte is ever exported, not even to stamp the key’s own identity onto itself. The mechanism is the subject of a signing key that never leaves KMS if you want to see how the signer is wired up.

Two things to get right. First, the AWS credentials: minting needs both kms:GetPublicKey and kms:Sign on the key, because it reads the public material and then signs the self-signature with it. The signer role you stood up in Parts 2 and 3 can do both; running this locally, your own credentials need the same. Second, and this is the same lesson Part 1 hammered on, pin --created. An OpenPGP fingerprint is derived partly from the creation time, so a different timestamp gives you a different fingerprint and, in effect, a different key as far as your tooling is concerned. Use the moment the KMS key was created and never let it drift: the key you embed in Part 5 and the key you publish here have to be byte-identical, and --created is what guarantees it.

Mint the rotation authority while you’re here

You won’t use it until Part 7, but the rotation-authority key is far easier to create now, alongside everything else, than to bolt on in a panic when you need it. It’s a break-glass key: an offline key whose only job is to vouch for a new signing key if the KMS one ever has to be replaced. The spare front-door key you tape to the back of a drawer, not the one on your keyring.

Because it’s break-glass, it isn’t a KMS key. You generate it on a trusted offline machine and the private half goes straight into cold storage:

gtb keys generate \
 --algorithm ed25519 \
 --name "Acme Rotation Authority" \
 --email release@acme.dev \
 --created "2026-06-02T00:00:00Z" \
 --output rotation-authority.asc

INFO Generated OpenPGP keypair algorithm=ed25519 public_output=rotation-authority.asc private_output=rotation-authority.priv.asc creation_time=2026-06-02T00:00:00Z fingerprint=...
WARN Move the private-half file to offline storage now. private_output=rotation-authority.priv.asc

For Ed25519 you get an armored public .asc and an armored secret-key block (.priv.asc, the same wire format gpg --export-secret-keys produces). Do what the warning says, and do it before you forget: move rotation-authority.priv.asc to offline storage immediately. An encrypted USB stick and a paper backup is not paranoid for a key you might not touch for two years and will desperately need when you do.

Notice it carries the same release@acme.dev user ID as the signing key, not a separate rotation@ address. That’s deliberate, and it’s the one place people trip: WKD groups keys by the email in their UID, so the matching address is what puts the rotation authority in the same bucket as the signing key, and into the same embedded trust set. Give it a different email and it quietly drops out of the published bucket, your embedded and WKD key sets stop matching, and the cross-check from Part 5 starts failing every update. Same release identity, two keys. The public half travels with the signing key from here on, published and embedded together.

Build the WKD tree

Now publish them. The way clients find a public key from an email address is the Web Key Directory: a fixed set of files under .well-known/openpgpkey/ on a web server, where each key lives at a path derived by hashing the local-part of its email. gtb keys wkd builds that tree for you, in pure Go, with no gpg or gpg-wks-client anywhere in sight:

gtb keys wkd \
 --domain acme.dev \
 --email release@acme.dev \
 --output ./wkd-staging \
 signing-key-v1.asc rotation-authority.asc

INFO WKD bucket email=release@acme.dev hash=... keys=2
INFO wrote path=wkd-staging/.well-known/openpgpkey/acme.dev/policy
INFO wrote path=wkd-staging/.well-known/openpgpkey/acme.dev/submission-address
INFO wrote path=wkd-staging/.well-known/openpgpkey/acme.dev/hu/...
INFO WKD tree complete output=wkd-staging method=advanced emails=1 files=3

Both keys carry release@acme.dev in their UID here, so they land in the same hu/ bucket, concatenated. (If you’d rather split them, give each a different email and pass --email twice.) The tree under ./wkd-staging holds a policy file (required by the spec, and empty), a submission-address file, and one hu/<z-base-32-hash> file per email with the keys inside.

One decision matters for where this gets served. The default --method is advanced, which serves the tree from a dedicated openpgpkey.acme.dev subdomain, which is why the path above has acme.dev nested inside it. Pass --method direct instead and the tree is served from acme.dev itself. Advanced is the modern default and what you want unless you’ve a reason otherwise; it does mean you’ll need DNS and TLS for openpgpkey.acme.dev.

Publish it somewhere the platform can’t reach

This is the bit that’s tempting to fudge, and the bit you mustn’t. Do not drop the WKD tree onto the same host, or under the same account, as your code and your releases.

Walk the attack through. Your binary will carry an embedded copy of the public key (Part 5). On every self-update it fetches the published key over WKD and checks the two agree before trusting a download. That cross-check is the entire defence. If an attacker who compromised your release platform could also rewrite the WKD tree, they’d swap both keys for one of their own, sign a malicious release with it, and the client would wave it straight through. The cross-check would be comparing a forged key against another copy of the same forged key. Worthless. The defence only holds if the published key and the embedded key come from infrastructure an attacker would have to breach separately. That argument is laid out in full in a signature the platform can’t forge; this is where you actually pay for it.

In practice that means a different host with its own credentials. A static host like Cloudflare Pages in Direct Upload mode does the job: you build the tree locally and push it with the Wrangler CLI under a token scoped to Pages edit and nothing else, no Git integration wired to your code repo. The token that can rewrite your keys has no power over your releases, and vice versa. For the advanced method you’ll also point DNS for openpgpkey.acme.dev at that host and let it terminate TLS. Whatever you pick, the test is simple: could one stolen credential change both the key in the binary’s update path and the key on the server? If yes, you’ve built a very elaborate way of trusting nobody.

Where this leaves you

You’ve turned a key locked inside KMS into a published, fetchable OpenPGP key without the private half ever surfacing, minted the offline rotation authority you’ll be glad of later, and put both somewhere your release platform can’t quietly rewrite. The verifying side of the loop now exists out in the world, ready to be checked against.

What’s missing is the checking. Part 5 bakes the public key into the binary as its trust anchor and turns enforcement on, so the tool refuses any update whose signature doesn’t hold, the moment that update lands.

Sign your own binaries with go-tool-base, part 3: keyless CI signing with OIDC

Fri, 19 Jun 2026 00:00:00 +0000

Part 2 left you with a KMS key your release pipeline can sign through and a role (<name>-signer) that’s allowed to call kms:Sign and nothing else. There’s one obvious question left hanging: how does a CI job become that role without an AWS access key stashed in a CI variable? Because a long-lived key sitting in your settings is exactly the kind of credential that ends up in a breach write-up. This part wires CI in with no stored credentials at all, on GitLab and GitHub.

The mechanism is OIDC federation, and the one-sentence version is: your CI platform already proves who it is to AWS, so let it. When a pipeline runs, GitLab or GitHub can hand the job a short-lived signed token (a JWT) that says “this is a tag pipeline on acme/acme-cli”. AWS trusts that token the same way a bouncer trusts a passport: it checks who issued it and what it says, and if the claims match a role’s trust policy, it swaps the token for temporary AWS credentials that live only for the job’s run. No key is stored anywhere; the credentials are minted on the spot and evaporate when the runner stops. The deep-dive on keyless CI covers the why and the threat model; here we just do the wiring.

Two things have to line up for this to work: the IAM OIDC identity provider (the thing in your AWS account that says “I trust tokens from this issuer”), and the signer role’s trust policy (which says “and only from these pipelines”). The terraform-aws-signing-kms module owns the second. The first comes from a sibling module.

Where the identity provider comes from

You don’t register the OIDC provider in the signing module. That’s deliberate: the same provider is shared across every role in the account that federates from CI, so it lives one level up, in terraform-aws-bootstrap. It takes a ci_provider input (github by default, or gitlab), registers the right IAM OIDC identity provider for that forge, and emits its ARN. You feed that ARN into the signing module. If you ran the bootstrap in Part 2 you already have it; if not, stand it up first and grab the oidc_provider_arn output.

That ARN is the only thing the two modules need to agree on. Everything else about which pipelines may sign lives in the signing module’s trust policy, which is what the rest of this part configures.

The GitLab path

Look the provider up by URL so you never hardcode the ARN, then pass it in along with the subject filter that scopes who can assume the role:

data "aws_iam_openid_connect_provider" "gitlab" {
 url = "https://gitlab.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn

 # oidc_issuer_host defaults to "gitlab.com"
 # oidc_audience defaults to "sts.amazonaws.com"

 ci_subject_filters = [
 "project_path:acme/acme-cli:ref_type:tag:ref:v*",
 ]

 key_administrator_arns = [/* ... from Part 2 ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

The interesting line is ci_subject_filters. GitLab stamps each CI token’s sub claim with the project path, the ref type and the ref. The pattern above reads as “tag pipelines on acme/acme-cli, for any ref starting v”. A branch pipeline or a merge-request pipeline carries ref_type:branch instead, so it simply doesn’t match, and the role refuses to be assumed. Your signer can only be driven from a release tag, which is the whole point: a dependency author opening an MR can’t trick CI into minting a signature.

On the pipeline side, the release job declares an id_tokens block so GitLab issues a token with the right audience, writes it to a file, and the AWS SDK picks it up:

goreleaser:
 rules:
 - if: '$CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/'
 id_tokens:
 AWS_WEB_IDENTITY_TOKEN:
 aud: sts.amazonaws.com
 variables:
 AWS_REGION: eu-west-2
 AWS_ROLE_ARN: ${SIGNER_ROLE_ARN}
 AWS_WEB_IDENTITY_TOKEN_FILE: /tmp/aws-token
 before_script:
 - echo "$AWS_WEB_IDENTITY_TOKEN" > /tmp/aws-token
 script:
 - aws sts get-caller-identity
 - goreleaser release --clean

AWS_ROLE_ARN plus AWS_WEB_IDENTITY_TOKEN_FILE is the convention the SDK recognises: it sees the two together and calls assume-role-with-web-identity for you, so by the time goreleaser runs it’s already the signer role. The actual signing job is Part 6; the aws sts get-caller-identity line is just a sanity check that federation worked. It should print the signer role’s ARN.

The GitHub path

Same shape, different issuer and a different sub format. GitHub’s sub support landed in module v0.1.2: earlier versions validated ci_subject_filters against GitLab’s format only and would reject a GitHub subject outright, so pin the version.

data "aws_iam_openid_connect_provider" "github" {
 url = "https://token.actions.githubusercontent.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 oidc_provider_arn = data.aws_iam_openid_connect_provider.github.arn
 oidc_issuer_host = "token.actions.githubusercontent.com"

 # oidc_audience still defaults to "sts.amazonaws.com"

 ci_subject_filters = [
 "repo:acme/acme-cli:ref:refs/tags/v*",
 ]

 key_administrator_arns = [/* ... */]
 automation_role_arn = data.aws_iam_role.automation.arn
}

Two differences from GitLab. The oidc_issuer_host has to change, because it’s the prefix on the trust-policy condition keys (token.actions.githubusercontent.com:sub rather than gitlab.com:sub). And the subject format is GitHub’s own: repo:<owner>/<repo>:ref:refs/tags/v* scopes the same way the GitLab pattern did, to tag refs only. The audience stays sts.amazonaws.com, because that’s the default aws-actions/configure-aws-credentials requests, so there’s nothing to override.

The workflow side is the official AWS action. It needs id-token: write permission to ask GitHub for the token in the first place, and contents: write so GoReleaser can create the release:

permissions:
 id-token: write
 contents: write
jobs:
 release:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: aws-actions/configure-aws-credentials@v4
 with:
 role-to-assume: ${{ secrets.SIGNER_ROLE_ARN }}
 aws-region: eu-west-2
 audience: sts.amazonaws.com
 - run: aws sts get-caller-identity
 - run: goreleaser release --clean

If you forget id-token: write, GitHub never issues a token and the action fails before it reaches AWS. It’s the most common GitHub-side trip-up.

The gotcha that wastes an afternoon: token audience

Here’s the one worth burning into memory, because the error message points you at the wrong thing. An IAM OIDC provider carries a client_id_list, and it rejects any token whose aud claim isn’t on that list before AWS even looks at the role’s trust policy. So if your CI token’s audience and the provider’s client ID don’t match, you get:

InvalidIdentityToken: Incorrect token audience

That reads like a role-trust problem, and you’ll go round in circles editing ci_subject_filters, but the trust policy was never consulted. The fix is to keep the audience aligned to sts.amazonaws.com everywhere: it’s the module’s oidc_audience default, it’s what terraform-aws-bootstrap puts on the provider’s client_id_list, it’s the aud in the GitLab id_tokens block, and it’s the default aws-actions/configure-aws-credentials requests. Leave all four alone and they agree by default. The moment you override one, override all of them, or you’ll meet that error.

Where this leaves you

There’s now no AWS access key anywhere in either forge. The pipeline proves it’s a release tag on your project, AWS hands it the signer role for the length of the run, and the credentials are gone the moment the runner stops. The signing module’s trust policy is the gate; the bootstrap module’s OIDC provider is the lock it hangs on. Both are verifiable in the public modules: terraform-aws-signing-kms for the role and terraform-aws-bootstrap for the provider.

The role can sign, but you still can’t verify anything yet, because nobody has the public half of that KMS key. Part 4 fixes that: gtb keys mint pulls the public key straight out of KMS, and gtb keys wkd publishes it somewhere the release platform can’t touch.

Sign your own binaries with go-tool-base, part 2: a signing key in AWS KMS

Wed, 17 Jun 2026 00:00:00 +0000

Part 1 left you with a working signing loop and one glaring weakness: the private key was a .pem on your laptop, and files get copied. This part fixes that. You’ll generate the production signing key inside AWS KMS, where it’s created and never comes out, and stand up a role that can sign with it. The key itself is the only thing we build here; wiring CI in over OIDC is Part 3.

The big idea is the same one the deep-dive spends its whole length on, so I’ll keep it short: you never hold the private key and you never sign with it. You ask KMS to sign on your behalf with kms:Sign, and the private half stays inside the HSM for its entire life. There is no export, no download, no “just this once” copy onto a runner. An attacker who owns your CI still can’t walk away with the key, because the key was never on the runner to begin with.

What you’ll need first

This part is OpenTofu (or Terraform, the module is fine with either). You’ll need:

An AWS account you can apply infrastructure into.
An IAM OIDC identity provider already registered in that account. KMS doesn’t need it, but the signer role we create trusts it, so it has to exist. If you haven’t got one, the sibling module terraform-aws-bootstrap provisions it (it’s the same family as the terraform-aws-security-baseline module). Its oidc_provider_arn output is exactly what we feed in below.

You don’t need a CLI, a pipeline or a public key yet. This is just the vault and the key that lives in it.

The module

The key, its alias, the signer role and the full key policy come from one public module, terraform-aws-signing-kms. Here’s the whole consumer block:

data "aws_iam_openid_connect_provider" "gitlab" {
 url = "https://gitlab.com"
}

module "signing_kms" {
 source = "gitlab.com/phpboyscout/signing-kms/aws"
 version = "0.1.2"

 name = "acme-release-signing-v1"
 description = "Acme release binary signing"

 oidc_provider_arn = data.aws_iam_openid_connect_provider.gitlab.arn
 ci_subject_filters = ["project_path:acme/acme-cli:ref_type:tag:ref:v*"]

 key_administrator_arns = [ /* operator role + account root */ ]
 automation_role_arn = data.aws_iam_role.automation.arn
}

# Carry these forward: Part 3 wires the signer role into CI, Part 4 mints
# the public key from the alias.
output "signer_role_arn" {
 value = module.signing_kms.signer_role_arn
}

output "signing_key_alias" {
 value = module.signing_kms.key_alias_name
}

A few of those values are doing more than they look, so let’s walk them.

name is acme-release-signing-v1, with the v1 on the end deliberately. The name derives the role (<name>-signer) and the alias (alias/<name>), and both of those want to outlive the key. When you rotate to a new key in Part 7 you’ll mint a -v2 and repoint things, so bake the version in now rather than wishing you had.

ci_subject_filters is in GitLab’s OIDC sub format here, and it’s the line that says which pipeline is allowed to assume the signer role: tag pipelines for any v* ref on acme/acme-cli, and nothing else (no branch builds, no merge requests). It’s the heart of Part 3, so I’ll leave the full explanation there. For now, know it’s not optional: an empty list would trust every token from the issuer, and the module refuses to let you do that.

key_administrator_arns and automation_role_arn are the two roles that manage the key, and the distinction between them matters. More on that in a moment.

tofu init
tofu apply

What you just built

A single asymmetric KMS key, plus the IAM scaffolding around it. The key is created with key_usage = SIGN_VERIFY and the spec RSA_4096, which is the module’s default and the one you want.

The obvious question, if you’ve signed things before, is why RSA-4096 and not Ed25519, which is smaller and faster. Two reasons, and neither is preference. The first is that AWS KMS simply doesn’t offer Ed25519 for asymmetric signing, so it’s off the table the moment you decide the key lives in KMS. The second is that OpenPGP, the format your signatures end up in, ties its packet encoding to the signing algorithm: the algorithm isn’t a detail you can swap underneath, it’s written into the bytes. RSA-4096 is the spec that satisfies both constraints, so it’s the secure default and you shouldn’t need to touch it.

Two more things to note about the key itself:

enable_key_rotation is off, and that’s intentional. AWS’s automatic yearly rotation only works on symmetric keys; an asymmetric SIGN_VERIFY key can’t be auto-rotated, because a new key would mean a new public half and every embedded trust anchor breaking at once. Rotation for signing keys is a deliberate, staged operation (mint a new key, publish it, repoint the alias), which is its own part later in the series.
The deletion window defaults to 30 days, the longest AWS allows. For a key this important, the longest possible “oops, undo” window is the safe choice.

The module also creates a stable alias, alias/acme-release-signing-v1. Always reference the key through its alias, never the raw key ID. The alias is what survives rotation: when v2 arrives, the alias gets repointed and everything calling through it keeps working.

Four principals, one key policy

This is the part worth slowing down for. A KMS key is governed by its key policy, and this module writes one policy that names four classes of principal, each with deliberately different reach:

The account root keeps kms:*. That’s AWS’s recommended break-glass: if every other path is locked out, the account owner can still recover.
The key administrators (key_administrator_arns, typically your operator role plus root) can administer the key, schedule its deletion, that sort of thing. They are not signers, and they can’t sign.
The automation role (automation_role_arn, the role your infra apply pipeline assumes) can manage the key as a Terraform resource, read it, tag it, even change its policy. What it deliberately cannot do is kms:Sign. Think about why: the role that applies your infrastructure runs on every change, so if owning that role let an attacker mint signatures, you’d have handed the whole point of the exercise back. Managing the key and using the key are two different powers, and only the signer gets the second one.
The signer role can call kms:Sign, kms:GetPublicKey and kms:DescribeKey, on this one key, and that’s the entire list. It can’t read other keys, can’t administer this one, can’t delete anything.

Here’s the detail that catches people out: the signer role has no attached IAM policy at all. Its permissions live entirely in the key policy. That’s not an oversight, it’s the design. One document, the key policy, is the single source of truth for who can do what to this key, so there’s no second place to check and no way for an attached role policy to drift out of sync with the key policy and quietly grant something nobody intended. If you want to know who can sign, you read one file.

The signer role is assumable via OIDC (those ci_subject_filters again), which is what lets a CI job step into it without any stored credential. That federation is Part 3’s whole job.

The outputs you’ll carry forward

The module hands back everything the later parts consume. The ones you’ll actually use:

key_id and key_arn, the bare ID and full ARN of the key.
key_alias_name and key_alias_arn, the stable alias. key_alias_name is the one you’ll pass to gtb as the KMS key reference, because it survives rotation.
signer_role_arn and signer_role_name, the role CI assumes to sign. signer_role_arn becomes an environment variable in your pipeline next part.

Keep these to hand. Part 3 needs signer_role_arn; Part 4 needs key_alias_name to mint the public key out of KMS.

What this costs

Worth a quick word, because this is the one part of the series that puts a line on an AWS bill. An asymmetric KMS key runs about a dollar a month, plus a tiny per-signature charge on the kms:Sign calls. For release signing, where you sign a handful of checksums files a month, the per-signature cost rounds to nothing. A dollar a month for a key that can’t be stolen off a laptop is the cheapest security control in this entire series.

Where this leaves you

The key exists, it lives somewhere it can never leave, and there’s a role that can sign with it (and a separate role that pointedly can’t). What there isn’t yet is any way for your release pipeline to become that role without a long-lived credential sitting in CI.

That’s the gap Part 3 closes: federating GitLab and GitHub into the signer role over OIDC, so a tagged release can assume it for the length of one job and nothing is stored anywhere. The ci_subject_filters line we glossed over here is where it starts.

Sign your own binaries with go-tool-base, part 1: sign and verify on your laptop

Mon, 15 Jun 2026 00:00:00 +0000

The quickest way to understand release signing is to do it once, by hand, with nothing but a key on disk. No cloud account, no CI, no cost. This first part of the signing series walks the whole loop on your laptop: make a key, sign a file, and verify the signature, including with plain gpg so you can see it isn’t locked to anything of ours. Everything you learn here maps one-to-one onto the AWS KMS workflow in the later parts; only where the key lives changes.

You’ll need the gtb CLI (installation docs). Make a scratch directory to work in, because we’re going to create a few files.

Make a signing key

gtb keys generate creates a keypair entirely inside the process and writes both halves to disk:

gtb keys generate \
 --algorithm rsa \
 --name "Acme Releases" \
 --email "release@acme.dev" \
 --created "2026-06-01T00:00:00Z" \
 --output signing.asc

INFO Generated OpenPGP keypair algorithm=rsa public_output=signing.asc private_output=signing.pem creation_time=2026-06-01T00:00:00Z fingerprint=...
WARN Move the private-half file to offline storage now. private_output=signing.pem

Two files come out: signing.asc (the public half) and signing.pem (the private half, a PKCS#1 PEM). The private half is the thing you guard. There’s no on-disk passphrase in this version of gtb, so keep it under filesystem encryption (LUKS, FileVault, or wrap it with age) rather than leaving it lying about.

One flag is doing quiet but important work: --created. An OpenPGP key’s fingerprint is derived partly from its creation time, so if you let it default to “now”, every run produces a different fingerprint. Pin it to a fixed instant and the key is reproducible, which matters the moment you start embedding it in a binary. Get in the habit now.

Mint the public key you’ll actually publish

You could hand signing.asc around as-is, but we’re going to produce the public key a slightly different way, with gtb keys mint:

gtb keys mint \
 --backend local \
 --key-id signing.pem \
 --name "Acme Releases" \
 --email "release@acme.dev" \
 --created "2026-06-01T00:00:00Z" \
 --output release.asc

INFO Minted OpenPGP key backend=local key_id=signing.pem output=release.asc creation_time=2026-06-01T00:00:00Z fingerprint=...

mint wraps a signing backend in OpenPGP framing and writes out the armored public key. Here the backend is local (a PEM file on disk), but in production it’ll be aws-kms pointing at a key you can’t hold. Minting the public key from the backend is the one habit worth forming early: it’s the only way to get the public half of a KMS key, so doing it the same way locally means the rest of the series is identical bar one flag. release.asc is the key you publish and embed from here on. (Because we pinned the same --created, its fingerprint matches the generated one exactly.)

Sign something

A real release signs its checksums.txt, so make a stand-in and sign it:

printf 'abc123 acme_linux_amd64\ndef456 acme_darwin_arm64\n' > checksums.txt

gtb sign \
 --backend local \
 --key-id signing.pem \
 --public-key release.asc \
 checksums.txt

INFO Signed file backend=local key_id=signing.pem public_key=release.asc input=checksums.txt output=checksums.txt.sig ...

That writes checksums.txt.sig, a detached, ASCII-armored OpenPGP signature. Note gtb sign takes --public-key: it cross-checks that the backend key matches the public key you claim to be signing as, and refuses if they diverge, so you can’t accidentally sign with the wrong key.

Verify it, two ways

First, the way your tool will do it on every self-update: against the public key. That path is the subject of a signature the platform can’t forge and we wire it into a real binary in part 5. For now, prove the signature is sound with something every machine already has, gpg:

gpg --import release.asc
gpg --verify checksums.txt.sig checksums.txt

gpg: Signature made ...
gpg: using RSA key ...
gpg: Good signature from "Acme Releases <release@acme.dev>"

Good signature is the whole point. The signature gtb sign produced is an ordinary OpenPGP detached signature, so anyone can verify it with the standard tool, no go-tool-base required. (gpg will warn the key isn’t certified in its web of trust; that’s expected and unrelated to whether the signature is valid.)

Now change a byte of checksums.txt and run the verify again. gpg reports BAD signature. That failure is the entire reason any of this exists: a tampered manifest no longer matches the signature, and a tool that requires a valid signature will refuse the update.

Where this leaves you

You’ve signed a file with a key you made and verified it independently. That’s the complete trust loop in miniature, and the shape never changes: a private key signs, a public key verifies, and the two are produced and checked the same way whether the private half is a .pem on your laptop or an HSM-held key in AWS.

The local key was the easy bit, and also the weakest: it’s a file, and files get copied. Part 2 moves the private key somewhere it can’t be copied at all, AWS KMS, and the only command that changes is the --backend flag.

Sign your own binaries with go-tool-base

Sat, 13 Jun 2026 00:00:00 +0000

If your CLI tool can update itself, it has a decision to make that nobody is watching: when it pulls down a new version, should it trust what just landed? A checksum tells it the bytes match a manifest. It does not tell it who wrote the manifest. Close that gap and your users get updates they can actually trust; leave it open and a compromised release host can hand them anything it likes. This series is the end-to-end “how”, using the signing tooling built into go-tool-base.

By the end you’ll have a CLI that ships releases signed by a key you control, verifies its own updates against that key, and does the whole thing with no gpg wrangling and no long-lived secrets sitting in CI. We did the why and the how it works in two deep-dives already, a signature the platform can’t forge and a signing key that never leaves KMS. This is the use-it counterpart.

What you’re protecting against

Nobody’s coming to clean your supply chain, so it’s worth being clear about the threat before you spend an afternoon on the fix. A checksum file sits next to the binary on the same release page. Whoever can swap the binary can swap the checksum in the same breath, and the hash still matches. A signature is different: it’s made by a private key the release platform never holds, and verified against a public key your tool fetches from somewhere the platform can’t reach. To forge a release that passes, an attacker would have to steal a key that, done right, was never anywhere they could get at it.

That “done right” is the whole series.

Two paths through it

You don’t need a cloud account to start. The series runs in two stages:

Learn it locally. Part 1 signs and verifies on your laptop with a plain key on disk. No AWS, no CI, no cost. It’s the fastest way to see every moving part for real.
Do it for production. Parts 2 onward move the private key into AWS KMS, where it’s generated and never leaves, and wire your release pipeline to sign through it over short-lived OIDC credentials.

Each part stands on its own and ends with something that works. They build in order, but you can stop after Part 1 with a genuinely useful skill and come back for the cloud parts when you need them.

Before you start

You’ll want a CLI built on go-tool-base to sign. If you haven’t got one, the Building a CLI with go-tool-base series gets you there in an afternoon; this one picks up where releases come in. You’ll also need the gtb CLI installed (the installation docs have the one-liner), and for the cloud parts, an AWS account and a GitLab or GitHub project to release from.

The parts

Sign and verify on your laptop: gtb keys generate, gtb sign, and gpg --verify, the whole loop with a local key.
A signing key in AWS KMS: stand up an asymmetric KMS key with the terraform-aws-signing-kms module.
Keyless CI signing with OIDC: federate GitLab and GitHub into the signer role, no stored credentials.
Mint and publish your public key: gtb keys mint from KMS, then gtb keys wkd to publish it off-platform.
Embed the key and require verification: bake the trust anchor into your binary and turn enforcement on safely.
Sign every release with GoReleaser: wire signing into a real tagged-release pipeline.
Rotation and break-glass: the part everyone skips, and how to do it without locking anyone out.

Start with Part 1. By the time you reach the end, the chain runs from a key born in a vault to a binary on a stranger’s machine checking, on its own, that the update it just fetched is really yours.

Generate a command from a script or a sentence with go-tool-base

Thu, 11 Jun 2026 00:00:00 +0000

You’ve got a Python script that already does the job. It’s sat in a tools/ directory somewhere, it works, and every few weeks someone copies it onto a laptop that doesn’t have the right version of pandas and it falls over. You’d like it to be a proper subcommand of your tool, a real Go binary you can ship, but porting it means the cobra wiring, the options struct, a test file, and a fight with the linter before any of it lands.

Or you don’t even have the script. You’ve just got a sentence in your head: “something that pings a list of URLs and tells me which ones are slow.” The logic is five minutes of thought; the boilerplate around it is the afternoon.

gtb generate command is built for exactly that gap. Hand it a script or hand it a sentence, and it writes the Go, the tests and the docs, then sends an autonomous agent through the result to make sure the thing actually builds, passes its tests and survives golangci-lint before it ever reaches your working tree.

Two ways in, the same files out

There are two flags, and they’re mutually exclusive:

--script <file> converts an existing bash, Python or JavaScript script.
--prompt "<text>" (or a path to a file) generates from a plain-English description.

Both land in the same place. A generated command called csv-stats gives you:

pkg/cmd/csv-stats/cmd.go: the cobra registration. This one is read-only; the generator owns it and will regenerate it.
pkg/cmd/csv-stats/main.go: the implementation, where your logic lives and where you’re free to edit.
pkg/cmd/csv-stats/main_test.go: a test file.
docs/commands/csv-stats/index.md: AI-written docs for the command.

The provider and model come from your config (ai.provider) or the --provider / --model flags. Everything below was generated with Claude Opus. We’ll take each in turn.

From a script: `csv_stats.py` becomes `csv-stats`

Here’s the script I want as a native subcommand. It reads a CSV and reports, per column, the row count, how many values are empty, and min/max/mean for the numeric ones. Nothing exotic, but enough that porting it by hand is a chore. Copy it into a file called csv_stats.py if you want to follow along:

#!/usr/bin/env python3
"""Summarise a CSV file's columns.

For every column it reports the row count and how many values are empty; for
columns whose values are numeric it also reports min, max and mean. A single
column can be selected with --column.

usage: csv_stats.py [--column NAME] <file.csv>
"""
import argparse
import csv
import sys


def is_number(value):
 """True if value parses as a float."""
 try:
 float(value)
 return True
 except (TypeError, ValueError):
 return False


def summarise(path, only_column=None):
 with open(path, newline="") as handle:
 reader = csv.DictReader(handle)
 if reader.fieldnames is None:
 print("error: empty CSV", file=sys.stderr)
 return 1

 columns = list(reader.fieldnames)
 if only_column is not None:
 if only_column not in columns:
 print(f"error: no such column: {only_column}", file=sys.stderr)
 return 1
 columns = [only_column]

 counts = {c: 0 for c in columns}
 nulls = {c: 0 for c in columns}
 numbers = {c: [] for c in columns}

 for row in reader:
 for c in columns:
 value = row.get(c, "")
 counts[c] += 1
 if value is None or value.strip() == "":
 nulls[c] += 1
 elif is_number(value):
 numbers[c].append(float(value))

 header = f"{'column':<20}{'count':>8}{'nulls':>8}{'min':>12}{'max':>12}{'mean':>12}"
 print(header)
 print("-" * len(header))
 for c in columns:
 nums = numbers[c]
 if nums:
 cmin = f"{min(nums):.2f}"
 cmax = f"{max(nums):.2f}"
 cmean = f"{sum(nums) / len(nums):.2f}"
 else:
 cmin = cmax = cmean = "-"
 print(f"{c:<20}{counts[c]:>8}{nulls[c]:>8}{cmin:>12}{cmax:>12}{cmean:>12}")
 return 0


def main():
 parser = argparse.ArgumentParser(description="Summarise a CSV file's columns.")
 parser.add_argument("csvfile", help="path to the CSV file")
 parser.add_argument("--column", help="only summarise this column")
 args = parser.parse_args()
 return summarise(args.csvfile, args.column)


if __name__ == "__main__":
 sys.exit(main())

One command points the generator at it:

gtb generate command \
 --name csv-stats \
 --short "Summarise CSV columns" \
 --script ./csv_stats.py

Your browser doesn't support embedded video; the demo converts csv_stats.py into a Go command and the repair agent builds, tests and lints the result.

What lands is not a transliteration. The Python kept everything in one function; the Go that came out is decomposed into named pieces, opens the file through the project’s injected filesystem (props.FS, an afero Fs) rather than os, and reports through the structured logger rather than print:

func summarise(fs afero.Fs, path, onlyColumn string) ([]string, error) {
	handle, err := fs.Open(path)
	if err != nil {
		return nil, errors.Wrapf(err, "failed to open CSV file %q", path)
	}
	defer func() {
		_ = handle.Close()
	}()

	reader := csv.NewReader(handle)
	reader.FieldsPerRecord = -1

	columns, indexByName, err := readColumns(reader, onlyColumn)
	if err != nil {
		return nil, err
	}

	stats := make(map[string]*columnStats, len(columns))
	for _, c := range columns {
		stats[c] = &columnStats{numbers: []float64{}}
	}

	for {
		record, readErr := reader.Read()
		if readErr != nil {
			if errors.Is(readErr, io.EOF) {
				break
			}

			return nil, errors.Wrap(readErr, "failed to read CSV record")
		}

		accumulate(stats, columns, indexByName, record)
	}

	return formatReport(columns, stats), nil
}

That decomposition, into readColumns, accumulate, formatReport, summaryValues and a couple of small formatting helpers, is the interesting part, and it didn’t come for free. The first thing the agent did after writing the code was build it, test it, and lint it. golangci-lint’s cyclop rule flagged a single fat summarise function well over its complexity ceiling of 10. So the agent read the file back, split the work into focused functions, and ran the checks again. It only stopped once the build, the tests and the linter were all clean. The tidy shape above is the agent arguing with the linter and winning, not the model’s first guess.

Then it just runs. In the demo I scaffolded the project without the init feature, so the tool reads sensible defaults and needs no config step, and csv-stats sample.csv prints real per-column counts, nulls and numeric stats (with the default features you’d run toolbox init, or pass --config, first). The full generated command, the three files and nothing else, is here: csv-stats-command.tar.gz.

From a sentence: a URL health-checker

No script this time. Just a description of the command I wish I had. --prompt takes a raw string, but a description with any detail to it is easier to read, and to keep, in a file, so I dropped it in healthcheck-prompt.txt:

Concurrently GET a list of URLs and report each one’s HTTP status and latency.

Flags:

--timeout: the per-request timeout

--file: read URLs from a file, one per line

--json: machine-readable output

Use httptest in the tests so they need no network.

The prompt describes what I want the command to do, including how the flags should behave. The flags themselves I declare up front with --flag (more on why that split matters below), and point the generator at the file:

gtb generate command \
 --name healthcheck \
 --short "Check URL health concurrently" \
 --flag "timeout:duration:per-request timeout:false:t:false:5s" \
 --flag "file:string:read URLs from a file, one per line" \
 --flag "json:bool:machine-readable output" \
 --prompt ./healthcheck-prompt.txt

Your browser doesn't support embedded video; the demo builds a concurrent URL health-checker from a natural-language description, then self-repairs until it builds clean.

And the flags feed straight in. RunHealthcheck reads the URL file from opts.File, the deadline from opts.Timeout, and the output format from opts.Json, then fans the requests out across goroutines, each writing into its own slot, exactly the way you’d write it by hand:

func RunHealthcheck(ctx context.Context, props *props.Props, opts *HealthcheckOptions, args []string) error {
	urls, err := collectURLs(props.FS, opts.File, args)
	if err != nil {
		return errors.Wrap(err, "failed to collect URLs")
	}

	if len(urls) == 0 {
		return errors.New("no URLs provided; pass URLs as arguments or via --file")
	}

	timeout := opts.Timeout
	if timeout <= 0 {
		timeout = defaultTimeout
	}

	client := &http.Client{Timeout: timeout}

	results := make([]result, len(urls))

	var wg sync.WaitGroup

	for i, u := range urls {
		wg.Add(1)
		go func(idx int, target string) {
			defer wg.Done()

			results[idx] = checkURL(ctx, client, target, timeout)
		}(i, u)
	}

	wg.Wait()

	return reportResults(props, opts.Json, results)
}

I asked for the tests to use httptest so they’d need no network, and they do. Each case spins up a local server, so go test is hermetic and the agent’s own test run during repair stays self-contained, and it wrote cases for the flags too, this one driving --json:

func TestRunHealthcheck_JSONOutput(t *testing.T) {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusNotFound)
	}))
	defer srv.Close()

	p := newTestProps()
	opts := &healthcheck.HealthcheckOptions{
		Timeout: 5 * time.Second,
		Json: true,
	}

	err := healthcheck.RunHealthcheck(context.Background(), p, opts, []string{srv.URL})
	if err != nil {
		t.Fatalf("unexpected error: %v", err)
	}
}

Same as before, it builds and runs: point it at a few URLs and it GETs them concurrently, reporting each status and latency. The full generated command is here: healthcheck-command.tar.gz.

What “self-repair” actually means

The agent isn’t a single shot at the model with a hopeful prompt. It’s a loop with real tools: it reads the project layout, reads the files it needs, and runs go build, go test and golangci-lint. When something fails, it reads the relevant code, rewrites it, and runs the checks again. It only declares success once all three pass with nothing outstanding. The repair agent’s instructions are deliberately blunt on that last point: a clean build and passing tests don’t count as done if the linter still has something to say.

A few flags shape how it runs:

--max-steps N raises the agent’s reasoning budget. The default is plenty for a command like these two, but a genuinely hairy conversion can run long, and this stops it stopping short.
--agentless skips the agent entirely and uses the older retry loop, if you’d rather keep the generation cheap and do the polishing yourself.
--non-interactive withholds the agent’s ability to ask you a question mid-run. It defaults on when the CI environment variable is set, so the thing never blocks a pipeline waiting for an answer that isn’t coming.

Flags you declare, logic it writes

The --timeout, --file and --json arrived as real flags on the command, but not because the prompt mentioned them. Flags are the generator’s job, not the prompt’s, and that split is deliberate. You declare each one with --flag (or the interactive wizard), as I did above, and the generator wires it onto the options struct and into the read-only cmd.go registration, which hands that struct straight to your Run function. The prompt is left to describe behaviour: what --timeout should bound, what --file should read, what --json should change.

So the agent, told exactly which option fields exist, wrote its logic against opts.Timeout, opts.File and opts.Json rather than inventing anything, and the finished command’s --help lists them with the 5s default and the -t shorthand I asked for. Leave the --flags off and it still works: the generator hands the agent an empty options struct, and it keeps those values as locals with sensible defaults, ready for a flag to be wired in later.

The one thing you don’t do is hand-edit cmd.go: it’s regenerated every time you add a flag or change the command, so reach for --flag, never the file. When a generation finishes, the quickest sanity check is the command’s own --help, which shows the flags it actually exposes.

One thing to keep in mind: the model isn’t deterministic. Run the same prompt twice and you’ll get two slightly different commands. If the first one isn’t quite right, regenerate, or nudge the prompt. Treat the output the way you’d treat a capable colleague’s first PR: read it, run it, and own what you merge.

And is it the best possible code, the best design? Probably not. That depends on the model you can afford to point at it, how much detail you put in the prompt, and a bit of luck on the day. What you can count on is a working starting point: something that builds, has tests, and uses proper Go idioms and the project’s own patterns, instead of a blank file and an afternoon of boilerplate. From there it’s yours to shape.

Where that leaves you

The generator does the boilerplate and has the argument with the linter so you don’t have to. What it can’t do is decide whether the command it built is the command you actually wanted. That part is still yours, which is rather the point. The full docs for both flags live in the AI conversion guide and the command generation reference, and they’re the place to go when you want the flags the prompt didn’t.

Building a web service with go-tool-base, part 6: seeing what your service is doing

Tue, 02 Jun 2026 00:00:00 +0000

On paper the macguffin service is finished. Part 5 left it typed, fast, documented and served over TLS. So you deploy it, traffic starts flowing, and a week later someone wanders over to say “it’s slow”.

Slow how? Slow where? Which endpoint, the gateway or the gRPC behind it, the store or the network in between? You open the logs and there’s a wall of “request completed” lines, no way to tell which ones belong to the same request, and not a word about the half-second one of them spent somewhere inside. The service is a black box that happens to be on fire.

That’s the gap this part closes. A long-running service needs to be observable: you need to see what it’s doing, how often, how fast, and where the time actually goes. And we get there without bolting on a separate stack, because go-tool-base speaks OpenTelemetry, and three signals fall out of the same wiring.

This part is written against go-tool-base v0.7.1, the release that brings the observability components in.

The three things you actually want to see

“Observability” is a grand word for three fairly down-to-earth questions. Logs answer what happened (you’ve had those since part 1). Metrics answer how often, and how fast: request rates, error counts, the spread of latencies. And traces answer where did the time go, by following a single request as it crosses from one piece of your system to the next.

For a service shaped like ours, that last one is the prize. A REST call hits the gateway, turns into a gRPC call, runs through the store, and comes back. A trace stitches that whole journey into one timeline, so “it’s slow” stops being a shrug and becomes “it’s slow in CreateMacguffin, in the store, for 400ms”. You can’t get there from a pile of disconnected log lines.

OpenTelemetry (OTel, if you’ve seen it bandied about) is the vendor-neutral standard for all three. You instrument once and ship the data to whatever backend you fancy (Jaeger, Grafana, Honeycomb, a plain collector) over one wire protocol, OTLP. go-tool-base does the instrumenting; you point it at a collector and pick your poison.

Turn it on

One call wires the lot. In the serve command from part 4, right after the controller exists:

// pkg/cmd/serve/main.go

import "gitlab.com/phpboyscout/go-tool-base/pkg/telemetry"

controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

// Build whichever signals are enabled, install them, flush them on shutdown.
if _, err := telemetry.Setup(ctx, p, controller); err != nil {
	return err
}

telemetry.Setup reads your config, builds whichever of the three signals you’ve switched on, and installs them as the process-wide OTel providers. It also hangs their shutdown on the controller, so when a SIGTERM lands the buffered spans and metrics flush before the process exits. That’s the same graceful-shutdown promise from part 1, now extended to your telemetry: a clean stop drops nothing, not even the trace that was mid-flight.

On its own that line shows you nothing. A signal you haven’t enabled is skipped, so an unconfigured service pays exactly nothing. You turn each on in config, which we’ll get to. First, the instrumentation.

Wire it onto the transports

Setup builds the pipeline; two short additions feed it. The gRPC server takes a stats handler:

// pkg/cmd/serve/main.go

grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger,
	gtbgrpc.OTelStatsHandler())

and the HTTP server takes a middleware. This is also where we finally meet go-tool-base’s middleware chain, which we’ve managed without until now. A chain is just an ordered stack of wrappers around your handler. We put the OTel one first so it opens the span, and the request logger second, so it runs inside that span and can see it:

// pkg/cmd/serve/main.go

chain := gtbhttp.NewChain(
	gtbhttp.OTelMiddleware("macguffin"),
	gtbhttp.LoggingMiddleware(p.Logger),
)

if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, mux,
	gtbhttp.WithMiddleware(chain)); err != nil {
	return err
}

That’s the whole code change. Both helpers, OTelMiddleware and OTelStatsHandler, lean on the standard OTel instrumentation libraries underneath (otelhttp and otelgrpc), so the spans and metrics they emit follow the conventions every backend already knows how to read. Every request now opens a span and records its own duration, on both transports, and you wrote none of it per handler.

Go deeper than the plumbing

The transport spans show the request crossing HTTP and gRPC, but they stop at the edge of your handler. To see where the time goes inside it, you open a span yourself, and that needs no go-tool-base API at all, just OpenTelemetry. Here’s the store doing it, opening a span around the work and tagging it with what it’s handling:

// internal/macguffin/store.go

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
)

var tracer = otel.Tracer("macguffinsvc/store")

func (s *Store) Create(ctx context.Context, name string, quantity int32) Macguffin {
	_, span := tracer.Start(ctx, "Store.Create")
	defer span.End()

	span.SetAttributes(
		attribute.String("macguffin.name", name),
		attribute.Int("macguffin.quantity", int(quantity)),
	)

	// ... store the macguffin, generate its id ...

	span.SetAttributes(attribute.String("macguffin.id", id))

	return m
}

The span starts as a child of whatever is already in the context, which is the gRPC server span the handler is running under, so it slots into the trace on its own. The one thing to get right is the context: give the store method a ctx context.Context first parameter and thread it down from the handler, and the span finds its parent. Tag it with the things you’d want when something’s wrong, the macguffin’s name and its id, and they show up on the span in the UI, right next to the timing.

Point it at a collector

The wiring’s on; now tell it where to send. The config lives under one root:

# config.yaml
telemetry:
 endpoint: "http://localhost:4318"
 insecure: true # plaintext: a local collector only
 tracing:
 enabled: true
 sampling: 1.0 # every trace while you're looking
 metrics:
 enabled: true
 logs:
 enabled: true

endpoint is your OTLP collector. Sampling defaults to a level-headed 0.1 in production (you rarely want every trace under real load), so turn it up to 1.0 while you’re developing or you’ll wonder where yours went. Stand up a collector (a local Jaeger handles traces and gives you a UI to poke at), start the service, and push some traffic through the REST side:

$ curl https://localhost:8443/v1/macguffins
$ curl -X POST https://localhost:8443/v1/macguffins -d '{"name":"the-grail","quantity":1}'

One request, the whole way through

Open the collector and there it is, the thing that justifies the exercise: a single trace for that POST, four spans deep. The HTTP server span on top, the gRPC call nested inside it, the gRPC handler inside that, and your Store.Create span at the bottom, with the macguffin’s name and id sat right on it. The REST request became a gRPC call, crossed the gateway we built in part 4, ran the handler and went into the store, and OTel followed it the entire way, because the trace context travels with the request and everything reads from the same providers. “It’s slow” finally has somewhere to point, and “slow where” has an answer.

The metrics ride the same pipeline: request counts, error counts, a latency histogram broken down by route and method. And your logs get an upgrade you didn’t ask for. Every “request completed” line now carries the trace_id and span_id of the request it describes, so a slow trace in the UI and its log line in your terminal share an id, and you can jump straight from one to the other. That correlation lands in your stderr logs and in the records shipped to the collector, both.

A word on the other telemetry

If you read Telemetry that asks first, you’ll know go-tool-base has a telemetry package for product analytics too: the opt-in kind that asks a user’s permission before it phones anything home. This is not that, and the difference is the whole point.

That analytics is about a user, and it’s gated behind informed consent: off until they say yes. This is about your service, emitted by you, to a collector you run, and it works on implied consent. You switched it on; there’s no end user in the loop to ask. Same package, deliberately separate gates, and never the twain shall meet. There’s a fuller post in that one distinction, and it’s the next thing I’m writing.

Where that leaves the service

Six parts in, the macguffin service starts cleanly, reports its health, serves gRPC and REST and interactive docs over TLS, shuts down without dropping a request, and now tells you what it’s doing while it runs, all from one annotated proto and a controller it registers against. Observability was the last missing sense. The thing was fast and correct before, but you couldn’t see inside it. Now you can, and it cost you about a dozen lines.

Building a web service with go-tool-base, part 5: docs that write themselves

Sat, 30 May 2026 00:00:00 +0000

The google.api.http annotations we added in part 4 have done one job so far: they told the gateway which REST calls map to which RPCs. But they describe the API precisely, the paths, the verbs, the request and response shapes, and a precise description of an API is most of an OpenAPI document. Feed that document to a viewer and you have an interactive docs site: every endpoint listed, every field typed, and a “try it” button that calls the real service.

So from one .proto we’ve had gRPC, then REST, and now documentation, none of it a separate thing to write or keep in sync.

The annotations, a third time

The pattern of this whole series is one source of truth and several outputs generated from it. The proto defined the gRPC service; the annotations on it generated the REST gateway; and those same annotations generate an OpenAPI document. Add a field to a message, and it shows up in the gRPC API, the REST API, and the docs, all at once, because all three are read from the proto.

Generate the OpenAPI document

This is one more buf plugin. A small wrinkle to know up front: grpc-gateway ships its own OpenAPI generator, but it emits OpenAPI v2 (the old Swagger format). For a v3 document we use kollalabs/protoc-gen-openapi, which reads the very same google.api.http annotations:

go install github.com/kollalabs/protoc-gen-openapi@latest

# buf.gen.yaml
plugins:
 # ... the go, go-grpc and grpc-gateway plugins from before ...
 - local: protoc-gen-openapi
 out: internal/docs/assets
 opt:
 - title=Macguffin API
 - default_response=false

go generate ./... now also writes an openapi.yaml. It’s the REST API described in full, drawn straight from the annotations:

# internal/docs/assets/openapi.yaml (generated, trimmed)
openapi: 3.0.3
info:
 title: Macguffin API
 version: 0.0.1
paths:
 /v1/macguffins:
 get:
 summary: ListMacguffins
 operationId: MacguffinService_ListMacguffins
 parameters:
 - name: pageSize
 in: query
 schema:
 type: integer
 format: int32
 responses:
 "200":
 description: OK
 content:
 application/json:
 schema:
 $ref: '#/components/schemas/ListMacguffinsResponse'

That GET /v1/macguffins operation is the get: "/v1/macguffins" rule from the proto, turned into OpenAPI. You wrote the annotation once; it now feeds three generators.

Serve it, viewer and all

A spec on disk is a means, not an end; people want to read it. The usual move is to bolt on a docs UI like Stoplight Elements or Swagger UI, which means vendoring a couple of megabytes of JavaScript into every project. go-tool-base’s openapi package does that part for you: the Stoplight Elements UI is embedded in the framework, so your project ships only its generated spec.

openapi.Register mounts both the document and the docs site onto a mux:

// internal/docs/docs.go

package docs

import (
	_ "embed"
	"net/http"

	"gitlab.com/phpboyscout/go-tool-base/pkg/openapi"
)

//go:embed assets/openapi.yaml
var spec []byte

// Register mounts /openapi.yaml and the Stoplight docs site (/docs/) onto mux.
func Register(mux *http.ServeMux) error {
	return openapi.Register(mux, spec, openapi.WithTitle("Macguffin API"))
}

The //go:embed bakes the generated spec into the binary (the same trick as part 3.5), and openapi.Register serves it at /openapi.yaml and the Stoplight site at /docs/. Wiring it into serve is one line, on the same mux the gateway is mounted on:

// pkg/cmd/serve/main.go (the existing HTTP wiring)

mux := stdhttp.NewServeMux()
mux.Handle("/v1/", gw) // REST, from part 4

if err := docs.Register(mux); err != nil {
	return err
}

Why it goes on the same server

That last point is doing more work than it looks. The docs, the spec, and the live REST API are all on the one HTTP server, so they share an origin. That’s what makes the “try it” console actually work: when you fill in a request in the docs and hit send, the browser calls /v1/macguffins on the same host it loaded the page from, with no cross-origin dance to configure. And because the certificate is your mkcert one, the page and its requests are all clean HTTPS, no warnings to click past, which is exactly why we set the local CA up back in part 2.

$ curl https://localhost:8443/openapi.yaml | head -4
# Generated with protoc-gen-openapi
openapi: 3.0.3
info:
 title: Macguffin API

Open https://localhost:8443/docs/ in a browser and there’s the service: every endpoint, every field, and a working console that calls the real thing.

One proto, three faces

Step back and look at what a single annotated .proto is now producing: a gRPC service for the things that speak it, a REST API for the things that don’t, and an interactive docs site for the people who have to consume either. One source, three faces, and nothing hand-maintained between them. That’s the whole argument for building it this way, and it’s why the annotations were worth the small ceremony.

The service is, by any reasonable measure, done: typed, fast, documented, and served over TLS. The last part is about what happens once it’s out there and taking traffic. In part 6 we add telemetry and logging, so you can see how it’s being used and why it’s slow, without bolting on a separate observability stack.

Building a web service with go-tool-base, part 4: REST for free, with the gateway

Fri, 29 May 2026 00:00:00 +0000

A quick tally of where part 3 left us. One domain, the Store. One gRPC service over it, mapping the domain to proto with toProto. And then a whole second transport, the REST layer, with its own routing and its own toDTO mapping the very same domain into the very same shape, by hand. Two encodings of one thing, drifting apart the moment anyone adds a field and forgets the other side.

I promised that doubling would go away. This is the part where it does, and the thing that does it is the grpc-gateway.

What the gateway actually is

The grpc-gateway is a reverse proxy, generated from your .proto, that speaks REST on the front and gRPC on the back. A JSON request comes in, the gateway turns it into the matching gRPC call, hands it to your gRPC server, and turns the gRPC response back into JSON on the way out.

Read that again with part 3 in mind, because it’s the whole point. The gateway does the JSON-to-proto-and-back encoding for you, using the proto types your gRPC server already produces. You wrote domain → proto once, in the gRPC adapter. The gateway supplies proto → JSON. There is no second hand-written encoding to keep in step, because there is no second implementation: REST becomes a generated front door onto the gRPC service you already have.

So the plan is short. Tell the proto which HTTP calls map to which RPCs, regenerate, wire the gateway in, and delete the part-3 REST layer entirely.

Map HTTP onto the proto

gRPC has no opinion about URLs and verbs; REST is all URLs and verbs. The bridge is an annotation, google.api.http, that you attach to each RPC to say “this one is GET /v1/macguffins/{id}”. Here’s the service with those rules added:

// proto/macguffin/v1/macguffin.proto

import "google/api/annotations.proto";

service MacguffinService {
 rpc GetMacguffin(GetMacguffinRequest) returns (Macguffin) {
 option (google.api.http) = {get: "/v1/macguffins/{id}"};
 }
 rpc ListMacguffins(ListMacguffinsRequest) returns (ListMacguffinsResponse) {
 option (google.api.http) = {get: "/v1/macguffins"};
 }
 rpc CreateMacguffin(CreateMacguffinRequest) returns (Macguffin) {
 option (google.api.http) = {
 post: "/v1/macguffins"
 body: "*"
 };
 }
}

Each rule is small but exact. {id} in the path binds to the id field of the request message. body: "*" on the create says the whole JSON body maps onto the request. The list takes no body and no path parameter, just the verb and path. This is the same information part 3’s hand-written routes carried, except now it lives next to the RPC it describes, and a generator reads it instead of you.

These rules go a good deal further than the three cases we need: query-string parameters, several URL bindings for a single RPC, custom verbs, choosing which field becomes the response body. When you reach for those, the grpc-gateway docs walk through the mapping, and the canonical reference is the HttpRule message that google.api.http comes from, its comments document every option.

The annotations.proto import comes from Google’s common protos, so tell buf where to find them by adding a dependency, then fetch it:

# buf.yaml
deps:
 - buf.build/googleapis/googleapis

buf dep update

Generate the gateway

This is another buf plugin, exactly like part 2’s. Install it:

go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest

and add it to the generators:

# buf.gen.yaml
version: v2
plugins:
 - local: protoc-gen-go
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-go-grpc
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-grpc-gateway
 out: internal/gen
 opt: paths=source_relative

go generate ./... now also writes macguffin.pb.gw.go, the gateway: a RegisterMacguffinServiceHandler function that, given a connection to your gRPC server, mounts the REST routes the annotations described.

Wire it on

The gateway needs to call your gRPC server, which means dialling it like any other client, over the same TLS, with credentials that trust its certificate. That’s fiddly to get right by hand, so go-tool-base’s gateway package does it for you. gateway.New opens the connection (matching your server’s transport security) and hands you a mux to register the generated handlers on:

// pkg/cmd/serve/main.go

package serve

import (
	"context"
	stdhttp "net/http"

	"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
	"google.golang.org/grpc"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	"gitlab.com/phpboyscout/go-tool-base/pkg/gateway"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	gtbhttp "gitlab.com/phpboyscout/go-tool-base/pkg/http"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	store := macguffin.NewStore()

	// gRPC: the one implementation, mapping the domain to proto.
	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	// REST, for free: the gateway proxies JSON/HTTP to the gRPC server above.
	gw, err := gateway.New(ctx, p.Config,
		func(ctx context.Context, mux *runtime.ServeMux, conn *grpc.ClientConn) error {
			return macguffinv1.RegisterMacguffinServiceHandler(ctx, mux, conn)
		})
	if err != nil {
		return err
	}

	mux := stdhttp.NewServeMux()
	mux.Handle("/v1/", gw)

	if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, mux); err != nil {
		return err
	}

	controller.Start()
	controller.Wait()

	return nil
}

The only macguffin-specific line is the one inside the callback, RegisterMacguffinServiceHandler. Everything around it, the dial, the credentials, the mux, is the framework’s. Mount the result under /v1/, register it on the same controller and HTTP server as before, and you’re done.

Delete the duplication

Here’s the satisfying bit. The hand-written REST adapter from part 3, the resthand package, the routes, the toDTO, all of it, comes out. You don’t need it: the gateway serves the same REST surface, backed by the gRPC service, from the proto. The serve command shrinks to one gRPC server and one gateway, and your codebase now has a single place where a macguffin becomes JSON.

See it work

The gateway answers REST, and it’s the same store the gRPC service uses:

$ curl https://localhost:8443/v1/macguffins
{"macguffins":[{"id":"m-1","name":"maltese-falcon","quantity":1}]}

$ curl -X POST https://localhost:8443/v1/macguffins -d '{"name":"the-grail","quantity":1}'
{"id":"m-2","name":"the-grail","quantity":1}

Create over REST, and the macguffin is there over gRPC a moment later, because both are the same implementation over the same Store:

$ grpcurl localhost:50051 macguffin.v1.MacguffinService/ListMacguffins
{ "macguffins": [ { "id": "m-1", ... }, { "id": "m-2", "name": "the-grail", ... } ] }

Errors, and changing them

Error handling comes across too. When a gRPC handler returns a status code, the gateway maps it to the matching HTTP status. The codes.NotFound we returned back in part 2 arrives as a 404, with a JSON error body, and we wrote none of it:

$ curl -s -o /dev/null -w '%{http_code}\n' https://localhost:8443/v1/macguffins/nope
404

That default mapping is the sensible one you’d reach for anyway. A few of the common codes:

gRPC code	HTTP
`InvalidArgument`	400
`Unauthenticated`	401
`PermissionDenied`	403
`NotFound`	404
`AlreadyExists`	409
`Internal`	500
`Unavailable`	503

So the rule of thumb is simply to return the right codes.* from your gRPC handlers, and the REST side gets the right status for free.

When the default shape isn’t what your clients expect, a {"error": {…}} envelope, a trace id header, a tweak to one particular status, you supply your own error handler. The grpc-gateway takes one as a runtime.ServeMuxOption, and gateway.New passes those straight through:

gw, err := gateway.New(ctx, p.Config, register,
	gateway.WithMuxOptions(runtime.WithErrorHandler(myErrorHandler)),
)

myErrorHandler receives the error and the http.ResponseWriter and writes whatever your API promises. That same WithMuxOptions hatch takes the gateway’s other knobs, header matchers, custom marshalers, and the rest; they’re all in the grpc-gateway docs.

Where this leaves us

This is the shape the series was building towards. One domain, one gRPC implementation, one domain → proto mapping you wrote by hand, and a REST API generated from the same proto that needed no second implementation and no second encoding. The things that speak gRPC get gRPC; the browser, the webhook and the curl get JSON; and there’s exactly one place to change when a macguffin grows a field.

Those google.api.http annotations have one more trick in them. They describe your REST API precisely enough to generate an OpenAPI document, and in part 5 we serve that as a live, clickable docs site, from the very same proto.

Building a web service with go-tool-base, part 3.5: the same server, now a website

Thu, 28 May 2026 00:00:00 +0000

The HTTP server from part 3 serves JSON. But net/http doesn’t care what you hand it: HTML, an image, a stylesheet, a whole little site, it’s all just bytes with a content type. So before we get back to the API in part 4, a short detour to prove the point and pick up a couple of genuinely useful tools: we’ll turn the macguffin service into a tiny website.

This is a bonus, off to the side of the API arc, but it earns its place. Real services nearly always grow a bit of HTML eventually: a status page, a landing page, a small admin view, an embedded docs site (we’ll do exactly that in part 5). The mechanics are the same every time, and worth having in hand.

A page from `html/template`

Go’s html/template renders HTML from a template and your data, and it escapes that data on the way out, so a macguffin called <script> becomes text rather than a problem. Here’s a page that lists the catalogue:

<!-- internal/site/templates/index.html -->
<!doctype html>
<html lang="en">
 <head>
 <meta charset="utf-8" />
 <title>Macguffins</title>
 <link rel="stylesheet" href="/static/style.css" />
 </head>
 <body>
 <h1>The macguffin catalogue</h1>
 <ul>
 {{range .}}
 <li>{{.Name}} <span class="qty">&times;{{.Quantity}}</span></li>
 {{end}}
 </ul>
 </body>
</html>

{{range .}} walks the slice we pass in, and {{.Name}} / {{.Quantity}} read each macguffin’s fields. The data is the same Store from part 2, so the page is a view onto the very same domain the gRPC and JSON APIs serve.

Shipping the files inside the binary

A template and a stylesheet are files, and you do not want to deploy a folder of loose assets next to your binary and hope they line up. Go’s embed package bakes them into the binary at build time, so the whole thing ships as one file.

// internal/site/site.go

package site

import (
	"embed"
	"html/template"
	"io/fs"
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

//go:embed templates static
var content embed.FS

var tmpl = template.Must(template.ParseFS(content, "templates/index.html"))

type Site struct {
	store *macguffin.Store
}

func New(store *macguffin.Store) *Site {
	return &Site{store: store}
}

func (s *Site) Routes() *http.ServeMux {
	static, err := fs.Sub(content, "static")
	if err != nil {
		panic(err) // the embedded path is a compile-time constant
	}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /{$}", s.index)
	mux.Handle("GET /static/", http.StripPrefix("/static/", http.FileServer(http.FS(static))))

	return mux
}

func (s *Site) index(w http.ResponseWriter, _ *http.Request) {
	if err := tmpl.Execute(w, s.store.List()); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Three things are doing the work. //go:embed templates static pulls both folders into the content filesystem. template.ParseFS parses the page from it once at startup. And http.FileServer(http.FS(static)) serves the stylesheet (and anything else under static/) straight from the embedded files, with content types set for you, so GET /static/style.css comes back as text/css.

The GET /{$} pattern is worth a note: the {$} anchors it to the exact root path, so / renders the page but /anything-else doesn’t accidentally match it.

If you’d rather edit templates without rebuilding during development, swap the embedded filesystem for the real one: http.FileServer(http.Dir("static")), and template.ParseGlob instead of ParseFS. Embed for release, disk for the edit-refresh loop; the handlers don’t change.

On the same server

Routes() hands back a *http.ServeMux, which is an http.Handler, so it registers exactly like the JSON API did, on the same controller, with the same TLS:

// pkg/cmd/serve/main.go (or a dedicated command)

if _, err := gtbhttp.Register(ctx, "site", controller, p.Config, p.Logger,
	site.New(store).Routes()); err != nil {
	return err
}

Because the certificate is the mkcert one from part 2, opening https://localhost:8443/ renders the page, stylesheet and all, with a clean padlock on any machine that trusts your local CA (where you ran mkcert -install); anywhere else, the browser flags the cert, exactly as it should.

The same hardened server, the same graceful shutdown, the same /healthz, now serving a website instead of (or alongside) JSON.

Back to the API

That’s the whole trick: the HTTP server is just net/http, and it will serve whatever you point it at, escaped and content-typed properly, shipped inside the binary. We’ll use exactly this in part 5 to serve interactive API docs.

Detour over. In part 4 we get back to the API and finally deal with that duplicated REST layer, the one we wrote twice and promised to delete.

Building a web service with go-tool-base, part 3: a REST service, by hand and by spec

Wed, 27 May 2026 00:00:00 +0000

The gRPC service from part 2 is the right core for service-to-service traffic. It is also useless to a great many of the things that might want to call it: a browser, a webhook from some SaaS, a partner who will not touch protobuf, a curl in a 2am runbook. They all want the same thing, plain JSON over HTTP.

So we need a REST face as well. The only way we know how so far is to build one, a second implementation of the macguffin service, this time over HTTP. We’ll do it two ways, by hand and from a spec, and wire it onto the very same controller from part 1. And yes, we’ll feel the duplication, because the next part is about making it disappear.

Why not just `net/http`?

A fair question before we register anything: go-tool-base’s HTTP server is net/http underneath, so why not stand up a http.Server{} yourself? You can, and nothing here hides the standard library from you. What gtbhttp.Register adds is the dull, easy-to-botch scaffolding that goes around the handler:

It’s still net/http. You write ordinary http.Handlers. There’s no bespoke router and no framework to learn underneath, just the standard library doing the serving.
Timeouts and limits you’d otherwise forget. A bare http.Server{} has no ReadTimeout or WriteTimeout, which is a Slowloris waiting to happen. You get sane read/write/idle timeouts and caps on header and body sizes.
The hardened TLS from part 2, the same shared certificate and the same 1.2-minimum AEAD config, with no extra setup.
Lifecycle for free. It registers on the same controller as your gRPC server, so a SIGTERM drains in-flight requests and stops both together, instead of you re-writing the signal-and-Shutdown dance for every service.
Health endpoints, /healthz, /livez and /readyz, backed by the controller and ready for an orchestrator to probe, with nothing to write.

You write the handlers; it owns the lifecycle and the hardening. With that settled, let’s write some handlers.

By hand, on the standard library

net/http’s ServeMux acquired method and path patterns in Go 1.22, so a small REST surface needs no router and no dependencies at all. And the hard part is already behind us: the Store from part 2 is our domain, and the HTTP handlers are just another thin adapter over it, exactly as the gRPC server was. A small type holds the store, and the routes hang off it:

// internal/resthand/server.go

package resthand

import (
	"encoding/json"
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

type API struct {
	store *macguffin.Store
}

func (a *API) Routes() *http.ServeMux {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /macguffins", a.list)
	mux.HandleFunc("GET /macguffins/{id}", a.get)
	mux.HandleFunc("POST /macguffins", a.create)

	return mux
}

Each pattern names a method and a path, and {id} is a named wildcard a handler reads back with r.PathValue("id"), no third-party router required. And because the domain Macguffin already carries JSON tags, this adapter can encode it straight to the response, with no separate type to map to:

// internal/resthand/server.go (same file)

func (a *API) get(w http.ResponseWriter, r *http.Request) {
	m, ok := a.store.Get(r.PathValue("id"))
	if !ok {
		http.Error(w, "macguffin not found", http.StatusNotFound)
		return
	}

	writeJSON(w, http.StatusOK, m)
}

list and create are the same shape: ask the store, encode the result with a small writeJSON helper. Routes() hands back a *http.ServeMux, which is itself an http.Handler, so it slots straight into the controller in a moment.

There’s nothing clever in any of it, and that’s the appeal: total control, no tooling, and because we serve the domain type directly, nothing to map. The cost is the kind that creeps up on you. Every route, every bit of marshalling, every status code is yours to write and keep correct, and as the surface grows, so does the area for small mistakes.

By spec, with oapi-codegen

The other road is the OpenAPI mirror of part 2’s proto. You describe the API in an OpenAPI document, and a generator turns it into a Go interface for you to implement. The tool is oapi-codegen.

First the contract, api/macguffin.openapi.yaml (trimmed to one path here):

# api/macguffin.openapi.yaml
paths:
 /macguffins/{id}:
 get:
 operationId: getMacguffin
 parameters:
 - name: id
 in: path
 required: true
 schema: { type: string }
 responses:
 "200":
 description: OK
 content:
 application/json:
 schema:
 $ref: "#/components/schemas/Macguffin"
 "404":
 description: Not found
components:
 schemas:
 Macguffin:
 type: object
 required: [id, name, quantity]
 properties:
 id: { type: string }
 name: { type: string }
 quantity: { type: integer, format: int32 }

Install the tool and tell it what to emit with a small config. We want the standard-library server, so std-http-server:

go install github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@latest

# api/oapi-codegen.yaml
package: restapi
output: internal/restapi/macguffin.gen.go
generate:
 std-http-server: true
 models: true

As in part 2, wire it into go generate so it can’t drift, this time sitting right alongside the buf directive:

// gen.go (at your module root): add the second line alongside part 2's

//go:generate buf generate
//go:generate oapi-codegen -config api/oapi-codegen.yaml api/macguffin.openapi.yaml

go generate ./... now regenerates both the gRPC and the REST code in one go. What oapi-codegen writes is the message models and, the important part, a ServerInterface, one method per operation:

// internal/restapi/macguffin.gen.go (generated)

type ServerInterface interface {
	ListMacguffins(w http.ResponseWriter, r *http.Request)
	CreateMacguffin(w http.ResponseWriter, r *http.Request)
	GetMacguffin(w http.ResponseWriter, r *http.Request, id string)
}

If that feels familiar, it should: it’s the exact shape from part 2. The generator hands you an interface and your job is a type that honours it, an adapter over the same Store. But here’s the wrinkle that earns this part its keep. oapi-codegen generated its own Macguffin type, a DTO, so the adapter has to map the domain into it. (The path parameter, at least, arrives already typed as id string rather than fished out by hand.)

// internal/restapi/server.go

package restapi

import (
	"net/http"

	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func (a *API) GetMacguffin(w http.ResponseWriter, _ *http.Request, id string) {
	m, ok := a.store.Get(id)
	if !ok {
		http.Error(w, "macguffin not found", http.StatusNotFound)
		return
	}

	writeJSON(w, http.StatusOK, toDTO(m))
}

// toDTO maps the domain type to oapi-codegen's generated Macguffin DTO.
func toDTO(m macguffin.Macguffin) Macguffin {
	return Macguffin{Id: m.ID, Name: m.Name, Quantity: m.Quantity}
}

That toDTO is the second mapping of this kind we’ve written. Part 2’s gRPC adapter had toProto; this one has toDTO. The same domain data, encoded twice, into two generated shapes, kept in step by hand. Hold that thought: part 4 is where that second mapping stops being something you write.

A generated helper turns your implementation into an http.Handler:

handler := restapi.HandlerFromMux(restapi.NewAPI(store), http.NewServeMux())

So which road? By hand when it’s a handful of endpoints you fully control and you’d rather not own a generator. Spec-first when the contract carries weight, several teams consume the API, the shape changes often, or you simply want the OpenAPI document to exist, which (spoiler) we’ll be serving as live, clickable docs in part 5.

Wire it onto the controller

Whichever road you took, you’re holding an http.Handler. We don’t replace the gRPC server from part 2; we add the HTTP one beside it, on the same controller, both reading from one shared store. This is the controller from part 1 doing exactly what it exists for: two transports, one lifecycle.

// pkg/cmd/serve/main.go

package serve

import (
	"context"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	gtbhttp "gitlab.com/phpboyscout/go-tool-base/pkg/http"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
	"gitlab.com/myorg/macguffinsvc/internal/resthand"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	// One shared domain behind both transports.
	store := macguffin.NewStore()

	// gRPC, from part 2.
	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	// HTTP, new this part. (Or the oapi-codegen handler; either is an http.Handler.)
	handler := resthand.New(store).Routes()
	if _, err := gtbhttp.Register(ctx, "http", controller, p.Config, p.Logger, handler); err != nil {
		return err
	}

	controller.Start()
	controller.Wait()

	return nil
}

The HTTP Register is the counterpart of part 2’s gRPC one: same controller, so a single SIGTERM drains and stops both together and /healthz reports on both. The HTTP server reads server.http.port, the gRPC server server.grpc.port from part 2, and both take their TLS from the shared server.tls block, the one certificate:

server:
 grpc:
 port: 50051
 reflection: true
 http:
 port: 8443
 tls:
 enabled: true
 cert: ./localhost+2.pem
 key: ./localhost+2-key.pem

A browser away

Here’s where the mkcert groundwork from part 2 pays off. Because that certificate is signed by a CA your machine already trusts, the service answers over HTTPS with no --cacert flag and, more to the point, no browser warning:

$ curl https://localhost:8443/macguffins
[{"id":"m-1","name":"maltese-falcon","quantity":1}]

$ curl https://localhost:8443/macguffins/m-1
{"id":"m-1","name":"maltese-falcon","quantity":1}

And /healthz now reports both transports, since they registered against the one controller:

$ curl https://localhost:8443/healthz
{"overall_healthy":true,"services":[{"name":"grpc","status":"OK"},{"name":"http","status":"OK"}]}

Open https://localhost:8443/macguffins in an actual browser and it just loads, green padlock and all. That matters more than it sounds, and it’s the reason we set the local CA up early: in part 5 the API docs are a web page, and a docs page behind a cert warning is a docs page nobody trusts.

Where this leaves us

Step back and count what we actually built. There’s one domain, the Store, and it didn’t change at all this part. What we added was a second delivery adapter over it: routing, marshalling, and, on the spec-first road, a toDTO mapping sitting beside part 2’s toProto. The same data, encoded into two generated shapes, kept in step by hand. That’s the real duplication, not the logic, but the transport scaffolding wrapped around it.

And it’s the itch the next part scratches. We built a second transport and a second encoding of the same domain, and kept them in step by hand. Part 4 is where that whole second layer, adapter and encoding both, stops being something you write at all. We felt the cost first on purpose; now we get to remove it.

Building a web service with go-tool-base, part 2: a gRPC service, with TLS

Tue, 26 May 2026 00:00:00 +0000

The heartbeat from part 1 runs, ticks along, and shuts down politely when you ask it to. It also talks to absolutely no one. A service people can actually call needs an API, and for a typed, fast, streaming-capable one, gRPC is the obvious first move.

The catch is that a production-grade gRPC server is rather more than grpc.NewServer(). You want health checks an orchestrator understands, reflection so you can poke at it without the .proto file in hand, a graceful shutdown that doesn’t guillotine calls that are still in flight, and TLS, which is where most people’s first attempt quietly goes wrong. The good news: part 1 already gave us the thing that carries all of it. A gRPC server is just another service to register on the controller.

Why gRPC, and not just REST

Worth a moment on why we’re reaching for gRPC at all, because for plenty of services a plain JSON-over-HTTP API is the right call and less faff. gRPC earns its place when a few of these matter:

A contract that’s enforced, not hoped for. The .proto is the single source of truth, and both ends are generated from it. You don’t hand-write JSON marshalling, and you don’t find out at runtime that the client and server disagree about a field’s type. Evolve the schema carefully (add fields by number) and old clients keep working.
Clients in any language, for free. The same .proto generates a Go server and a Python, TypeScript, Rust or Java client with nobody writing an SDK by hand. For an internal service that several teams call, that one point can decide it.
It’s built for service-to-service traffic. Binary protobuf is smaller and quicker to encode than JSON, calls multiplex down a single HTTP/2 connection, and streaming (from the client, the server, or both at once) is a first-class thing rather than something you bolt onto REST with websockets.
Deadlines, cancellation and a health protocol come built in, rather than conventions you reinvent for every service.

The trade-offs are real. A browser doesn’t speak gRPC natively, and a binary protocol is fiddlier to poke at than a JSON endpoint you can curl (which is exactly why reflection and grpcurl exist). That’s not a reason to avoid it; it’s the reason this series doesn’t stop at gRPC. In part 4 we put a REST/JSON face on this very service, so the things that call it get the typed, fast core and the things that can’t speak gRPC still get a friendly HTTP surface. You don’t have to pick a side.

Define the contract

gRPC starts with a schema. Here’s a small macguffin service, macguffin.proto:

// proto/macguffin/v1/macguffin.proto

syntax = "proto3";

package macguffin.v1;

option go_package = "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1;macguffinv1";

service MacguffinService {
 rpc GetMacguffin(GetMacguffinRequest) returns (Macguffin);
 rpc ListMacguffins(ListMacguffinsRequest) returns (ListMacguffinsResponse);
 rpc CreateMacguffin(CreateMacguffinRequest) returns (Macguffin);
}

message Macguffin {
 string id = 1;
 string name = 2;
 int32 quantity = 3;
}

message GetMacguffinRequest { string id = 1; }
message ListMacguffinsRequest { int32 page_size = 1; }
message ListMacguffinsResponse { repeated Macguffin macguffins = 1; }
message CreateMacguffinRequest { string name = 1; int32 quantity = 2; }

From proto to Go

If gRPC in Go is new to you, this is the part that catches people out: you don’t write the Go for those messages and that service interface, you generate it from the .proto. The proto is the source of truth; a compiler turns it into Go you import and build against. Same goes for a client in any other language, all from the same file.

That compiler is protoc, and on its own it’s a faff. You install it, then a separate plugin for each output you want (protoc-gen-go for the message types, protoc-gen-go-grpc for the client and server stubs), keep their versions in step, and drive the lot with a command line of -I include paths and --*_out flags that’s easy to get subtly wrong.

buf is the friendlier way to run exactly that. It wraps protoc and its plugins behind a couple of small config files, handles the plugin versions, and turns that gnarly invocation into a single word. It’s become the usual way to work with protobuf in Go, and it’s what we’ll use here.

At a minimum you need three binaries on your PATH: buf itself, and the two plugins it drives. go install is the quickest way to get them:

go install github.com/bufbuild/buf/cmd/buf@latest
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Those land in $(go env GOPATH)/bin, so make sure that’s on your $PATH. Then describe what you want generated in a buf.gen.yaml:

# buf.gen.yaml
version: v2
plugins:
 - local: protoc-gen-go
 out: internal/gen
 opt: paths=source_relative
 - local: protoc-gen-go-grpc
 out: internal/gen
 opt: paths=source_relative

Each part of that earns its place. version: v2 is buf’s config format. The plugins list names the generators to run, and we run two, because gRPC in Go arrives in two halves:

protoc-gen-go turns the messages into Go structs, the Macguffin type and the request and response types, in a macguffin.pb.go file.
protoc-gen-go-grpc turns the service into the client and server scaffolding, in a macguffin_grpc.pb.go file, including the MacguffinServiceServer interface you’re about to implement.

out: internal/gen is where the files land, and paths=source_relative lays them out mirroring the proto’s own folders (so proto/macguffin/v1/... becomes internal/gen/macguffin/v1/...) rather than deriving the path from the go_package line. Then run it:

buf generate

Both files appear under internal/gen/macguffin/v1, and we’re ready to write the implementation.

Running that by hand once is fine; remembering to run it every time the .proto changes is where it goes wrong, and the generated code quietly drifts out of step, usually right before a demo. Wire it into go generate instead. Drop a one-line directive in a file at your module root, say gen.go:

// gen.go (at your module root)

package macguffinsvc

//go:generate buf generate

Now go generate ./... regenerates everything from the proto, and it’s the same one command for any other generator you add later. Run it whenever the schema changes, and in CI if you want to catch a stale checkout.

If OpenAPI is your map of the territory

If your mental model of an API contract is an OpenAPI (Swagger) document, a .proto is the same idea wearing fewer clothes: a typed, language-neutral description of a service that both ends generate from. The thing you notice first is how much less of it there is. Here’s that Macguffin message again, and the same shape written as an OpenAPI schema:

message Macguffin {
 string id = 1;
 string name = 2;
 int32 quantity = 3;
}

Macguffin:
 type: object
 properties:
 id:
 type: string
 name:
 type: string
 quantity:
 type: integer
 format: int32

And that pattern holds across the whole service. The proto above, three calls and five messages, is about twenty lines. Describe the same surface in OpenAPI and you’re closer to a hundred, because OpenAPI also pins down the HTTP verbs, paths, status codes and content types: the transport details a proto leaves out on purpose. That isn’t OpenAPI being bloated; it’s describing more. But when the contract is the thing you care about, the proto says it with less ceremony, and it doesn’t wed your API to HTTP, which is exactly what lets us serve this same service over gRPC now and REST later. (We’ll generate a real OpenAPI document from this proto in part 5, for the readers who still want one.)

Implement it

Generating the code gave you the message types and, more to the point, an interface to satisfy. Open macguffin_grpc.pb.go and you’ll find MacguffinServiceServer, one method per RPC in the proto:

// internal/gen/macguffin/v1/macguffin_grpc.pb.go (generated)

type MacguffinServiceServer interface {
	GetMacguffin(context.Context, *GetMacguffinRequest) (*Macguffin, error)
	ListMacguffins(context.Context, *ListMacguffinsRequest) (*ListMacguffinsResponse, error)
	CreateMacguffin(context.Context, *CreateMacguffinRequest) (*Macguffin, error)
	mustEmbedUnimplementedMacguffinServiceServer()
}

That interface is the server-side contract. Each method takes the request message you defined and hands back the response message, plus an error. Writing the type that honours it, the actual logic behind each call, is the part that’s yours: the proto pins down the shape of the conversation, and this is what the service actually does when one happens.

The one curious line is mustEmbedUnimplementedMacguffinServiceServer(). Alongside the interface, buf generated an UnimplementedMacguffinServiceServer struct with a do-nothing stub for every method, and you embed it in your own type. It earns its keep twice over. It satisfies that unexported method, so your type counts as a real implementation. And it future-proofs you: add a new RPC to the proto later and your existing server still compiles, falling back to the stub (which returns a clean “unimplemented” error) until you write the real method.

Before we satisfy that interface, one separation worth making up front. The gRPC server is a delivery mechanism, not the place the data lives. If we stuff the map of macguffins straight inside it and then build an HTTP server next part, we’d have two servers each hoarding their own copy of the same data. So keep the domain, the macguffins and what you can do with them, in its own type, and let each transport be a thin layer over it.

Here’s that domain: an in-memory store standing in for the repository a real service would have. Nothing in it knows about gRPC, HTTP, or any wire format.

// internal/macguffin/store.go

package macguffin

import "sync"

// Macguffin is the domain type. The JSON tags will let a hand-written HTTP
// handler serve it directly in part 3.
type Macguffin struct {
	ID string `json:"id"`
	Name string `json:"name"`
	Quantity int32 `json:"quantity"`
}

type Store struct {
	mu sync.Mutex
	items map[string]Macguffin
	seq int
}

func (s *Store) Get(id string) (Macguffin, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()

	m, ok := s.items[id]

	return m, ok
}

List and Create are the same shape, and NewStore seeds it with a single maltese-falcon. Now the gRPC server is thin: it embeds the stub, holds a store, and each method calls the store and translates the result into the generated protobuf type.

// internal/grpcsvc/server.go

package grpcsvc

import (
	"context"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

type Server struct {
	macguffinv1.UnimplementedMacguffinServiceServer

	store *macguffin.Store
}

func New(store *macguffin.Store) *Server {
	return &Server{store: store}
}

func (s *Server) GetMacguffin(_ context.Context, req *macguffinv1.GetMacguffinRequest) (*macguffinv1.Macguffin, error) {
	m, ok := s.store.Get(req.GetId())
	if !ok {
		return nil, status.Errorf(codes.NotFound, "macguffin %q not found", req.GetId())
	}

	return toProto(m), nil
}

// toProto maps the domain type to the generated protobuf DTO.
func toProto(m macguffin.Macguffin) *macguffinv1.Macguffin {
	return &macguffinv1.Macguffin{Id: m.ID, Name: m.Name, Quantity: m.Quantity}
}

ListMacguffins and CreateMacguffin are the same: call the store, map the result. The one habit worth keeping is to return real gRPC status codes (codes.NotFound here) rather than bare errors, so callers get something they can branch on.

That toProto step is worth a second look, because it comes back round later. The domain has one shape, the proto has its own generated Macguffin, so the adapter maps between them. It’s a small price for a single transport. In part 3 we add a second transport with its own generated type and pay that price again, and part 4 is where we stop paying it twice.

Wire it onto the controller

This is the part that earns its keep. First, generate a serve command, the same way the CLI series generated its commands:

gtb generate command \
 --name serve \
 --short "Run the macguffin service"

That scaffolds two files: pkg/cmd/serve/cmd.go (generated, and wired into your command tree for you) and pkg/cmd/serve/main.go, which holds a RunServe function for your logic. Fill it in:

// pkg/cmd/serve/main.go

package serve

import (
	"context"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	gtbgrpc "gitlab.com/phpboyscout/go-tool-base/pkg/grpc"
	"gitlab.com/phpboyscout/go-tool-base/pkg/props"

	macguffinv1 "gitlab.com/myorg/macguffinsvc/internal/gen/macguffin/v1"
	"gitlab.com/myorg/macguffinsvc/internal/grpcsvc"
	"gitlab.com/myorg/macguffinsvc/internal/macguffin"
)

func RunServe(ctx context.Context, p *props.Props, _ *ServeOptions, _ []string) error {
	controller := controls.NewController(ctx, controls.WithLogger(p.Logger))

	store := macguffin.NewStore()

	grpcSrv, err := gtbgrpc.Register(ctx, "grpc", controller, p.Config, p.Logger)
	if err != nil {
		return err
	}

	macguffinv1.RegisterMacguffinServiceServer(grpcSrv, grpcsvc.New(store))

	controller.Start()
	controller.Wait()

	return nil
}

That’s the whole server. gtbgrpc.Register does four things in one call: it builds a *grpc.Server, wires the standard gRPC health service to the controller’s health reports (the ones we met in part 1), registers Start, Stop and Status against the controller so the lifecycle is handled, and hands you back the server to register your own service on, which is the RegisterMacguffinServiceServer line. After that it’s the same Start() / Wait() we used for the heartbeat.

It reads its port from config (server.grpc.port, falling back to server.port), so a minimal config is:

server:
 grpc:
 port: 50051
 reflection: true

Poke it

Build, run mytool serve, and reach for grpcurl. Reflection is on, so you don’t need the .proto to hand:

$ grpcurl -plaintext localhost:50051 list
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection
macguffin.v1.MacguffinService

$ grpcurl -plaintext localhost:50051 macguffin.v1.MacguffinService/ListMacguffins
{
 "macguffins": [
 { "id": "m-1", "name": "maltese-falcon", "quantity": 1 }
 ]
}

And the health service is already answering, wired straight to the controller, without you registering a thing:

$ grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
{ "status": "SERVING" }

That’s the lifecycle work from part 1 paying out: the controller’s health is the gRPC health, and a SIGTERM still drains and stops the server cleanly.

Now turn on TLS

Here’s the bit people brace for. Plaintext gRPC is fine on a laptop and unacceptable the moment it leaves one. With go-tool-base it’s a config change, not a code change.

The fiddly part of local TLS is usually the certificate. A hand-rolled self-signed one means passing a -cacert to every client and clicking past browser warnings. mkcert makes that go away: it creates a local certificate authority and installs it into your system’s (and your browser’s) trust stores, so anything it signs is simply trusted. Set the CA up once:

mkcert -install

Then mint a certificate for the names the service answers on:

mkcert localhost 127.0.0.1 ::1

That writes localhost+2.pem (the certificate) and localhost+2-key.pem (the key), signed by your now-trusted local CA. Doing this properly now pays off later: in part 3 the HTTP server, and in part 5 the API docs in a browser, both lean on that certificate being trusted with no warning.

Point the tool’s config at the pair, under the shared server.tls block:

server:
 grpc:
 port: 50051
 reflection: true
 tls:
 enabled: true
 cert: ./localhost+2.pem
 key: ./localhost+2-key.pem

No code changes. Run mytool serve again and it comes up over TLS:

INFO starting gRPC server tls=true addr=:50051

Because the certificate is signed by a CA your machine already trusts, the client needs no extra flags:

$ grpcurl localhost:50051 macguffin.v1.MacguffinService/GetMacguffin -d '{"id":"m-1"}'
{ "id": "m-1", "name": "maltese-falcon", "quantity": 1 }

A plaintext client is now refused, as it should be. (In production you’d point those same two config keys at whatever your real CA issues; the wiring doesn’t change.) Two details are worth knowing about what just happened, because both are easy to get wrong by hand. The server uses a hardened TLS config (1.2 minimum, AEAD cipher suites, X25519), so you’re not accidentally shipping a weak handshake. And the listener advertises HTTP/2 over ALPN, the h2 protocol gRPC rides on, which sounds like a footnote until you discover that recent gRPC clients refuse a TLS connection that doesn’t offer it. The framework sets it for you; it’s the single most common reason a hand-rolled gRPC-over-TLS server works with old tooling and mysteriously rejects a current client. All of that lives in the shared pkg/tls package.

I put the certificate under server.tls rather than server.grpc.tls deliberately. That shared block is the cert every transport falls back to, so the HTTP server in the next part and the transports after it can use the same one, with a per-transport override only where you actually need it.

The short version

A few files in, you have a real gRPC API: a typed contract, an implementation, health an orchestrator understands, reflection for poking, a clean shutdown, and TLS, and the only part that was actually yours to write was the service logic. The reference for the server helpers is the gRPC component doc, and the add-a-gRPC-service how-to has the manual-wiring path if you ever want it.

Next part puts an HTTP face on the very same controller, REST handlers and the same health endpoints an orchestrator probes, sharing that one certificate.

Building a web service with go-tool-base, part 1: lifecycle and graceful shutdown

Mon, 25 May 2026 00:00:00 +0000

A CLI does its job and gets out of the way. You run it, it prints something or writes a file, the process exits, done. Then one day you want the opposite: a thing that stays running. A server answering requests, a worker chewing through a queue, something that sits there until you tell it to stop. And the moment a process is long-lived, a pile of fiddly problems lands on your desk that a short-lived command never had to think about.

How does it shut down when Kubernetes sends it a SIGTERM, without dropping the requests it’s halfway through? How does anything outside it know it’s alive, or ready for traffic? When one part falls over at 3am, does the whole thing come down, or pick itself back up? None of that is your actual service. It’s the plumbing around it, and it’s the sort of plumbing that’s easy to write almost right and only notice the gap in during an incident.

This is a new series, a companion to the one on building a CLI. That series gets you a working command-line tool; this one turns it into a web service, a piece at a time: gRPC, HTTP, a gateway that bridges the two, TLS across all of them, and live API docs. But every one of those is a long-running process, so we start with the part they all stand on.

Same shape as before, each part stands alone. By the end of this one you’ll have a process that starts cleanly, reports its own health, and shuts down without dropping anything, and you won’t have written the lifecycle code yourself. The series is written against go-tool-base v0.6.0, the release that brings the web-service components in.

What every long-running process needs

go-tool-base’s answer to all of the above is the controls package, and its centrepiece is the Controller. You hand it a set of services, things with a way to start, a way to stop, and a way to report health, and it owns their lifecycle. It starts them, watches for the operating system asking the process to quit, drives a graceful shutdown in the right order, and keeps a running picture of whether everything is alright.

A “service” here is deliberately loose. An HTTP server is one. A gRPC server is one. So is a background worker that wakes every few seconds, or a queue consumer. The controller doesn’t care what’s inside; it cares that it can start it, stop it, and ask after its health. That’s the whole trick: get those three verbs right once, in one place, and everything you bolt on later inherits them.

A service in thirty lines

Let’s build the smallest useful one: a heartbeat that logs a tick every second. It isn’t exciting, but it’s a real long-running service, and it shows every moving part without a transport getting in the way.

// main.go

package main

import (
	"context"
	"os"
	"sync/atomic"
	"time"

	"gitlab.com/phpboyscout/go-tool-base/pkg/controls"
	"gitlab.com/phpboyscout/go-tool-base/pkg/logger"
)

func main() {
	log := logger.NewCharm(os.Stderr, logger.WithTimestamp(true))

	// The controller owns the process lifecycle: it starts registered services,
	// watches for SIGINT/SIGTERM, and drives a graceful shutdown.
	controller := controls.NewController(context.Background(), controls.WithLogger(log))

	var beats atomic.Int64

	controller.Register("heartbeat",
		// Start launches the work. The context is cancelled when the controller
		// shuts down, so the goroutine just watches ctx.Done().
		controls.WithStart(func(ctx context.Context) error {
			go func() {
				ticker := time.NewTicker(time.Second)
				defer ticker.Stop()

				for {
					select {
					case <-ctx.Done():
						return
					case <-ticker.C:
						log.Info("beat", "count", beats.Add(1))
					}
				}
			}()

			return nil
		}),
		// Stop runs during shutdown for any explicit cleanup.
		controls.WithStop(func(_ context.Context) {
			log.Info("heartbeat stopping", "total_beats", beats.Load())
		}),
		// Status reports health. Here we're healthy as long as we're ticking.
		controls.WithStatus(func() error { return nil }),
	)

	controller.Start()
	controller.Wait()
}

A few things are earning their keep there. NewController takes a context and some options, here just a logger. Register names a service and gives it its three verbs through functional options: WithStart launches it (and is handed a context that gets cancelled when the controller shuts down, which is the hook the goroutine watches), WithStop is called during shutdown for cleanup, and WithStatus answers “are you alright?”. Then controller.Start() launches everything and controller.Wait() blocks until the whole thing has stopped. The three options are all in pkg/controls.

Build it and run it:

go run .

INFO beat count=1
INFO beat count=2
INFO beat count=3

A service, running. Now for the half that’s easy to get wrong.

Shutting down on purpose

Press Ctrl-C, or send the process a SIGTERM the way an orchestrator would, and watch what it does:

WARN [Controller] : received signal signal=terminated
WARN [Controller] : Stopping Services
INFO heartbeat stopping total_beats=3
INFO [Controller] : Stopped

Nothing in our thirty lines handled a signal. The controller registered its own handlers for SIGINT and SIGTERM, and when one arrived it cancelled the context that every service’s Start is watching, gave them a window to finish, ran each Stop, and exited cleanly. That cancel-the-context-then-Stop order is the thing: it’s exactly what stops an HTTP server from dropping requests it’s mid-way through when the pod rolls. We’ve got it here for a heartbeat that does nothing on the way out, and we’ll get the same order for free for every real transport we add later.

You can tune the window with WithShutdownTimeout, and turn the signal handling off entirely (handy in tests) with WithoutSignals. The defaults are the right ones for a service in a container.

Health, before anything’s asking

That third verb, WithStatus, is the start of the health story, and it’s worth seeing now even though nothing’s calling it yet. The controller can report three separate things: overall status, liveness, and readiness. Those aren’t the same question. Liveness is “is this process wedged and in need of a restart”; readiness is “should traffic come here yet”. An orchestrator uses them differently, which is why they’re kept apart, and a service can answer them separately by adding WithLiveness and WithReadiness alongside WithStatus.

Right now nothing asks, because we’ve no transport. But this is the quiet payoff of putting lifecycle first: when we add an HTTP server in part 3, these reports are what back its /healthz, /livez and /readyz endpoints, and when we add gRPC in part 2 they back the standard gRPC health service, with no re-plumbing on our side. The controller also carries a restart policy for services that should pick themselves back up, and standalone health checks for things like “can I still reach the database”, but those earn their place once we’ve something worth checking.

Where this leaves us

A few lines in, we’ve a process that starts, ticks along, answers for its own health, and stops cleanly when the platform asks it to, on a controller that the real transports will register against unchanged. The heartbeat is a stand-in. Next part we swap it for a proper gRPC service, give it TLS, and the controller barely notices the difference, which is the entire point of it.

If you want to read ahead, the controls component has the full interface, and the service-orchestration deep-dive covers how the startup ordering and shutdown actually work underneath.

Building a CLI with go-tool-base, part 5: a CLI that updates itself

Sun, 24 May 2026 00:00:00 +0000

You ship version one. A week later someone finds a bug, you fix it, you cut version two. Now for the awkward part: how does the person who installed version one ever get version two? Email them? Hope they wander back to the install page? For a CLI that lives on people’s machines, “go and re-download it” is the answer that quietly strands half your users on old, broken builds. This part closes that gap, and like most of this series, the work is already done for you: your tool has shipped with an update command since part 1.

As before, this is written against go-tool-base v0.6.0 (gtb version).

The command is already there

update is one of the default features, so it’s been in your binary all along. Your users run:

mytool update

and the tool fetches the newest release, checks it, and replaces itself in place. No package manager, no re-download, no instructions. The rest of this part is about what that one command actually does, and how to make sure the binary it pulls down is the one you shipped.

Where it looks for releases

A tool can’t update itself without knowing where its releases live. That’s the --repo you passed back in part 1: it filled in your tool’s release source, the platform, owner and repository it checks. For --repo myorg/mytool that’s github.com/myorg/mytool, and mytool update looks at that project’s releases.

go-tool-base speaks more than one platform here, GitHub, GitLab, Gitea, Codeberg, Bitbucket, or a plain HTTP server, so the same command works whether you publish on github.com or your own GitLab. If you ever need to point somewhere else (a mirror, a private host), the custom release source how-to covers it; for a private repository it reads a token the same way the rest of the tool does.

What `update` does, step by step

When a user runs it, the command walks a short, careful path:

Resolve the latest release from your release source.
Compare versions. It reads the version baked into the running binary and compares it, as semver, against the latest. If you’re already current, it says so and stops: already running latest version, v1.2.0. (If your build somehow reports a version ahead of the latest published, it tells you off in character: your tardis travelled too far into the future...)
Download the right archive for the user’s OS and architecture.
Verify it before trusting it (the next section).
Replace the running binary with the new one, in place.
Bring the config along. If your tool has the init feature (it does by default), the update then runs the new binary’s init over the user’s config directory to fold in anything the release added.

That last step is easy to miss and matters more than it looks. A new version often ships new config: a key for a feature you just added, a changed default. Rather than leave the user a version behind, with code that expects settings their config file has never heard of, update re-runs init against their existing config once the swap is done, non-interactively (it passes --skip-login --skip-key, so nobody gets re-prompted for a token). It’s the same initialiser system from part 2, reused: the merge keeps what the user set and adds what the new version introduced, so the binary and its config move forward together. Turn the init feature off and this step is simply skipped, there’s no config to keep in step with.

There are two flags worth knowing. --version v1.3.0 targets a specific release instead of the latest, handy for pinning or rolling back. And --force updates even when the version check thinks you don’t need to. Most of the time, a bare mytool update is the whole story.

Downloaded isn’t the same as trusted

A binary that arrives over the network is a binary you didn’t build on the machine it’s running on, and a self-updater that swaps itself for whatever the server sent is a lovely way to ship a corrupted or tampered build straight into your users' hands. So before the swap, update verifies what it downloaded against a checksum manifest, the checksums.txt GoReleaser produces alongside your binaries. If the hash of the downloaded archive doesn’t match the one in the manifest, the update aborts and nothing gets replaced.

By default this is best-effort: a release that ships a checksums.txt is verified, but a release without one is updated with a warning rather than a hard stop. When you want the guarantee, make it mandatory:

# in your tool's config
update:
 require_checksum: true

Now a missing or mismatched checksum is a refusal, not a shrug. I wrote up why this matters, and exactly what it does and doesn’t buy you, in verifying your own downloads.

The limit is worth stating plainly, because it’s the whole reason there’s a “part two” to this story. A checksum proves the binary matches the manifest on the same release page. It catches a corrupted download or a botched upload cold. What it cannot catch is an attacker who owns the release platform and swaps both the binary and its checksum in the same breath, because then the two still agree. Closing that gap needs a signature whose trust root the release host can’t reach, which is a different piece of machinery (and a post of its own). go-tool-base now does exactly that: self-update signature verification has shipped, the binary checking a detached signature against a key it both embeds and fetches over WKD (how it works). Until you turn signing on for your own tool, checksums are the floor, and a worthwhile one.

Seeing it work without publishing anything

Here’s the catch with writing about self-update: you can’t update from a release you haven’t published, and your tutorial tool isn’t on anyone’s GitHub. There’s a flag for exactly this, meant for offline and air-gapped installs but perfect for a look under the hood: --from-file installs from a local release archive instead of the network.

Build a snapshot of your tool the way your release pipeline would (GoReleaser’s --snapshot builds the archives without publishing), then point update at one:

goreleaser release --snapshot --clean
mytool update --from-file ./dist/mytool_Linux_x86_64.tar.gz

You’ll watch the same extract-and-swap the network path uses, with nothing published and no release source involved. It’s also genuinely useful in its own right, for shipping into environments that can’t reach the internet.

The real loop

In production the cycle is the one part 1 already set you up for. The project gtb scaffolds ships a GoReleaser config and a release pipeline, so the flow is:

Tag a version and push the tag.
CI builds the binaries for every OS and architecture, generates checksums.txt, and publishes them as a release on your source.
Your users run mytool update and get it, verified.

You write git tag v1.3.0 && git push --tags; everyone who installed v1.2.0 is one command away from the fix. That’s the whole point of putting the update channel inside the tool: shipping a fix becomes tagging a release, and nothing else.

What this buys you

A tool that updates itself turns “please go and reinstall” into mytool update, and a tool that verifies what it updates to turns “I hope that download was clean” into a checked guarantee. Both came with the scaffold; the only work was understanding them. The full reference, including the config keys and the per-platform release sources, is in the update command docs and the auto-update concepts page.

Next part is the last one, and it’s about what happens after your tool is out there doing its job: telemetry and logging, so you can see how it’s actually being used without spying on the people using it. Until then, tag a release and watch your tool catch up to itself.

Building a CLI with go-tool-base, part 4: an AI dungeon master

Sat, 23 May 2026 00:00:00 +0000

I run a Dungeons & Dragons game on the odd weekend, so when I sat down to put an AI feature inside a CLI, my first instinct wasn’t a chatbot. It was: could the tool run a little adventure, with an AI as the dungeon master? It turns out that’s a near-perfect way to learn the chat client, because the thing that makes a game trustworthy, rules the players can’t break, is exactly the thing that makes any AI feature trustworthy. So this part builds mytool adventure: a tiny dungeon you play in your terminal, narrated by an AI that is firmly on a leash.

Part 3 pointed AI at your CLI from the outside (an agent driving your commands over MCP). This part goes the other way: AI inside your tool, as a feature you write. The worry everyone has about that is fair, AI output is unpredictable, and a CLI is meant to be dependable. The whole lesson here is how you square those two: you don’t hope the model behaves, you box it in with rules it can’t escape and mechanics it doesn’t get to invent.

As before, this is written against go-tool-base v0.6.0 (gtb version).

Behind the DM screen

A turn of our game looks like this: the player types what they want to do, the AI dungeon master narrates what happens and offers a few choices, and round it goes until the adventure reaches an end. The trick is where the truth lives. The model’s job is the prose, and only the prose. Everything else is yours:

The rules live in the system prompt: what the DM may and may not do.
The mechanics live in Go functions the model calls as tools (dice, combat). It never makes a number up.
The state lives in a Go struct you hand the model fresh every turn, so it never has to remember, and can’t quietly rewrite history.
The shape of each turn is a typed Go struct the model fills in, so your code always gets back something it can render, never a wall of prose to parse.

Two go-tool-base capabilities do the heavy lifting: the AI calling your Go functions, and the AI handing back a typed struct instead of text you have to regex. The game is just a fun excuse to use both at once.

Wiring a provider

The chat client (pkg/chat) is a library you import; you don’t need any special feature flag for it. It does need an API key, and it’ll find one from a few places. The simplest, for now, is the well-known environment variable for your provider:

export ANTHROPIC_API_KEY="sk-ant-..." # or GEMINI_API_KEY, OPENAI_API_KEY

That’s the bottom of the client’s lookup chain, which is fine for playing locally. For a tool you actually ship, go-tool-base has the ai feature and its mytool init wizard (the same initialiser system from part 2) to store the key properly, and there’s a whole post on where a CLI should keep your keys. For learning the client, an env var is plenty.

Scaffold the command

You know this step from part 1:

gtb generate command --name adventure --short "Play a dungeon adventure"

Everything below goes in the RunAdventure function the generator left you in pkg/cmd/adventure/main.go, plus a couple of types and helpers in the same package.

The state is yours, not the model’s

Start with the truth. The game state is a plain Go struct that you own. The model never holds it; instead you hand it the current state at the top of every turn (more on that in the loop). This is the part to grow: start small, then add rooms, items, NPCs, quest flags, whatever your adventure needs. Nothing else in the design has to change when you do.

// GameState is the single source of truth for the game. Extend it freely.
type GameState struct {
	PlayerHP int
	Location string
	Inventory []string
	Foes map[string]int // foe name -> remaining hit points
}

// summary renders the state into a line the model is given each turn.
func (g *GameState) summary() string {
	foes := make([]string, 0, len(g.Foes))
	for name, hp := range g.Foes {
		foes = append(foes, fmt.Sprintf("%s (%d HP)", name, hp))
	}
	return fmt.Sprintf("You have %d HP, at %s, carrying %s. Foes: %s.",
		g.PlayerHP, g.Location, strings.Join(g.Inventory, ", "), strings.Join(foes, ", "))
}

And the shape of a turn, the thing the model has to produce:

type Turn struct {
	Narration string `json:"narration"`
	Choices []string `json:"choices"`
	GameOver bool `json:"game_over"`
}

The dungeon master’s tools

A tool in pkg/chat is a chat.Tool: a name, a description the model reads to decide when to use it, a parameter schema, and a handler. The handler gets the model’s arguments as raw JSON and returns any value (which the framework JSON-encodes back to the model) or an error.

The simplest possible one is a die roll. This is the canonical “give the model something it’s bad at” tool, because language models cannot be trusted to roll fairly or even add up:

func rollTool() chat.Tool {
	return chat.Tool{
		Name: "roll",
		Description: "Roll a die with the given number of sides; returns 1..sides.",
		// Use an anonymous struct so the schema's properties sit at the top level,
		// which is where SetTools looks. A named type would hide them behind a $ref.
		Parameters: jsonschema.Reflect(struct {
			Sides int `json:"sides" jsonschema:"description=number of sides on the die"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			var a struct {
				Sides int `json:"sides"`
			}
			if err := json.Unmarshal(args, &a); err != nil {
				return nil, err
			}
			if a.Sides <= 0 {
				a.Sides = 20
			}
			return rand.Intn(a.Sides) + 1, nil
		},
	}
}

That comment about the anonymous struct matters, by the way. Reflect a named type and jsonschema emits a top-level reference with the real fields tucked inside, and the tool ships with no parameters at all. An anonymous struct inlines them where the framework expects. It’s the one sharp edge in the whole exercise.

Combat is where state actually changes, so combat is a tool too. Note it takes the foe by name and looks it up in Foes, so it works for the goblin and for any creature you add later, without touching this function:

func attackTool(game *GameState) chat.Tool {
	return chat.Tool{
		Name: "attack",
		Description: "Resolve the player's attack on a named foe. Rolls to hit, applies damage.",
		Parameters: jsonschema.Reflect(struct {
			Target string `json:"target" jsonschema:"description=the name of the foe being attacked"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			var a struct {
				Target string `json:"target"`
			}
			if err := json.Unmarshal(args, &a); err != nil {
				return nil, err
			}
			hp, ok := game.Foes[a.Target]
			if !ok {
				return map[string]any{"error": "no such foe: " + a.Target}, nil
			}
			if rand.Intn(20)+1 < 10 {
				return map[string]any{"hit": false, "foe": a.Target}, nil
			}
			dmg := rand.Intn(6) + 1
			hp -= dmg
			if hp < 0 {
				hp = 0
			}
			game.Foes[a.Target] = hp
			return map[string]any{
				"hit": true, "foe": a.Target, "damage": dmg,
				"foe_hp": hp, "defeated": hp == 0,
			}, nil
		},
	}
}

A bad target comes back as a plain error string, which the framework hands to the model so it can recover (apologise, pick a real foe) rather than crash.

That’s the whole tool set, and there’s deliberately nothing here for reading the state. The model never fetches it. Instead the loop hands it the current state at the top of every turn, which we wire up shortly. A language model has no memory you can rely on, so rather than trust it to remember the fight, you give it the truth each time.

The turn is a tool too

Here’s the neat part. The chat client won’t let a single call both run tools and return a typed struct, they’re separate modes. So instead of asking for the struct afterwards, we make submitting the turn into a tool of its own. The dungeon master ends its turn by calling submit_turn, and its handler captures the typed Turn into a variable we hold:

func submitTurnTool(out *Turn) chat.Tool {
	return chat.Tool{
		Name: "submit_turn",
		Description: "End your turn. Call this exactly once, last, with the turn's outcome.",
		Parameters: jsonschema.Reflect(struct {
			Narration string `json:"narration" jsonschema:"description=two-sentence narration of what just happened"`
			Choices []string `json:"choices" jsonschema:"description=the actions the player may take next"`
			GameOver bool `json:"game_over" jsonschema:"description=true only if the game has ended"`
		}{}),
		Handler: func(_ context.Context, args json.RawMessage) (any, error) {
			if err := json.Unmarshal(args, out); err != nil {
				return nil, err
			}
			return "turn recorded", nil
		},
	}
}

So the turn’s structure is enforced by a schema, same as any other tool’s parameters. Your loop gets a populated Turn every round, never prose.

The rules

This is where you bound the model. The system prompt is the rulebook, and it leans hard on the tools so the DM has no room to freelance the mechanics:

const dmRules = `You are the dungeon master of a short fantasy adventure. Each turn
you are given the current game state and the player's action.

Resolve the action and end the turn:
- If the player attacks, you MUST call the attack tool with the foe's name to
 resolve it. Do not decide the hit or the damage yourself.
- For any other chance event, call the roll tool and use its result.
- For simple actions, just narrate them.
- Then call submit_turn exactly once: a two-sentence narration, two or three
 choices, and game_over.

Trust the state you are given; never contradict it. A foe at 0 hit points is dead
and stays dead. The game ends when the player's hit points reach 0 (they lose), or
when the player reaches a satisfying ending. When it ends, set game_over and narrate
the finish.

Keep the tone light and quick.`

Two of those lines carry the weight. Trusting the state you are given, and never contradicting it, is what keeps the world consistent: the state is handed in fresh every turn (the next section), so the model works from the truth instead of from a memory it does not reliably have. And you MUST call the attack tool is what stops it quietly deciding hits and damage itself when it would rather just narrate. Those two are the difference between a game with rules and a model telling a story.

The loop

Now stitch it together. Create the client with the rules as its system prompt, register the tools once, and run a turn each time the player acts:

func RunAdventure(ctx context.Context, props *props.Props, opts *AdventureOptions, args []string) error {
	game := &GameState{
		PlayerHP: 20,
		Location: "the mouth of a damp cave",
		Inventory: []string{"a short sword", "a guttering torch"},
		Foes: map[string]int{"goblin": 12},
	}
	var turn Turn

	client, err := chat.New(ctx, props, chat.Config{
		SystemPrompt: dmRules,
		MaxSteps: 8, // roll/attack, then submit_turn
	})
	if err != nil {
		return err
	}

	if err := client.SetTools([]chat.Tool{
		rollTool(),
		attackTool(game),
		submitTurnTool(&turn),
	}); err != nil {
		return err
	}

	action := "I step into the cave."
	for {
		turn = Turn{}
		// Hand the model the current truth, then the player's action.
		input := fmt.Sprintf("State: %s\nThe player: %s", game.summary(), action)
		if _, err := client.Chat(ctx, input); err != nil {
			return err
		}

		fmt.Println("\n" + turn.Narration)
		if turn.GameOver {
			return nil
		}
		action, err = chooseAction(turn.Choices)
		if err != nil {
			return err
		}
	}
}

The same client runs every turn, so the conversation and the tools carry through the whole game; and the State: line you prepend is always current, because the attack tool mutated game last turn. The model is never trusted to remember, only to narrate.

The one helper I glossed is chooseAction. A bare fmt.Scanln would do, but we can do much better with almost no effort, and make a point while we’re at it. The framework already leans on Charm’s huh for its init wizard, you met it in part 2, so we’ll use the same library for a proper menu, with one deliberate addition:

func chooseAction(choices []string) (string, error) {
	const other = "__other__"

	opts := make([]huh.Option[string], 0, len(choices)+1)
	for _, c := range choices {
		opts = append(opts, huh.NewOption(c, c))
	}
	opts = append(opts, huh.NewOption("Something else...", other))

	var pick, custom string
	form := huh.NewForm(
		huh.NewGroup(
			huh.NewSelect[string]().
				Title("What do you do?").
				Options(opts...).
				Value(&pick),
		),
		// A second step that only appears when the player chose "Something else".
		huh.NewGroup(
			huh.NewInput().
				Title("Describe your action").
				Value(&custom),
		).WithHideFunc(func() bool { return pick != other }),
	)
	if err := form.Run(); err != nil {
		return "", err
	}

	if pick == other {
		return custom, nil
	}
	return pick, nil
}

The select gives the player a tidy arrow-key menu instead of typing a number, but the addition that earns its keep is the last option. “Something else…” is always there, and choosing it unfolds a second step (huh shows or hides a group with WithHideFunc) where the player types whatever they actually want to do. That free text goes straight to the dungeon master as the next turn’s input, and because the DM is an AI bound by the rules rather than a switch statement over three fixed choices, it just copes. Bargain with the goblin, search your pockets, set the cave alight: the model narrates it within the rules you gave it, rolling and applying damage through the same tools. That is the agency a scripted game can’t offer, and it’s the natural place to start building your own richer interactivity on top.

Play it

Set your key, build, and go:

export ANTHROPIC_API_KEY="sk-ant-..."
just build
./bin/mytool adventure

A turn looks like this (your wording will differ every time; the mechanics won’t):

You swing your short sword at the goblin, the blade whistling through the damp cave
air. The creature snarls as it tries to dodge your blow.

What do you do?
> Attack the goblin again
 Try to push deeper into the cave
 Retreat to the entrance
 Something else...

Your blade whistles through the air, but the nimble goblin dances back just in
time. It lunges forward with a rusty dagger in return, yet its clumsy strike only
finds empty air.

What do you do?
> Swing your sword again!
 Try to intimidate the creature
 Retreat from the cave
 Something else...

Behind that, the dungeon master called attack each turn (a hit, then a miss), the goblin’s hit points changed in the GameState you own, and the next turn handed that updated state straight back to the model. The prose is the model’s; every number is yours.

The pattern under the game

Strip the dungeon away and you’re left with the thing worth keeping. An AI feature you can ship is one where you’ve kept the model away from everything that has to be right: the rules live in the system prompt, the mechanics in typed Go tools the model must call, the state in a struct you hand it fresh each turn rather than trust it to remember, and the output in a struct it fills in rather than free text. Do that and the model’s unpredictability is confined to exactly where you want it, the wording, and walled out of everywhere you don’t, the maths, the state, the shape of the result.

Two honest limits worth knowing. There’s no temperature dial on the client (the setting that would let you turn the model’s randomness down), so you can’t make the prose reproducible; you make the mechanics reproducible instead, which for most features is what you actually needed. And a tool calling loop is several round-trips to the model per turn, so it’s not free, keep MaxSteps tight for anything interactive.

That’s the foundation, and the state struct is already sized for more than one fight: it carries a location, an inventory and a map of foes you’ve barely touched. Add a move tool that updates Location, a use_item tool that reaches into Inventory, a second creature in Foes, even a give_quest flag, and the adventure grows without the architecture changing. The model just gets more tools to call and more truth to read. Saved games come nearly free, too: the client can snapshot and resume a conversation. Next part leaves AI behind and gets the tool ready to look after itself: shipping signed self-updates, so a new release reaches your users safely. Until then, go explore the cave.

Building a CLI with go-tool-base, part 3: expose your CLI to AI agents

Fri, 22 May 2026 00:00:00 +0000

“Make it work with AI” is the request that lands on your desk with a thud and no further detail. The first time it landed on mine I braced for a treadmill of integration work: an adapter for this assistant, a wrapper for that one, one per client, forever. Then I looked at the hello command we built back in part 1. It has a name, a one-line description, and (once you give it some) typed, documented flags. That is exactly the shape an AI agent needs to call a tool. You already did the hard part.

This part wires that up: turning the CLI you’ve been building into something an AI assistant can drive, with no AI code of your own. The how-it-works behind it is in your CLI is already an AI tool; here we just use it.

A version note, as in the earlier parts: this is written against go-tool-base v0.6.0 (gtb version). The tool is young and moving, so if you’re on a newer release and a command or its output has shifted, check there first.

The translator you already have

An AI agent that wants to call local tools needs three things: a list of named operations, a description of each so it knows when to reach for them, and a typed parameter schema for each so it knows how to call them. A good CLI is already all three. The only missing piece is a translator between “this is a CLI” and “this is a set of tools an AI can call”, and that translator is the Model Context Protocol (MCP), an open standard every serious assistant now speaks.

Your tool already ships it. mcp is one of the default features, so it’s been in your binary since you scaffolded in part 1, no flag required. Check:

./bin/mytool mcp --help

You’ll see subcommands you never wrote. The rest of this part is just three of them.

See what the agent sees

Before you connect anything, look at what your tool would expose. This writes the tool definitions to a file:

./bin/mytool mcp tools

{
 "tools": [
 {
 "name": "mytool_hello",
 "description": "Say hello",
 "inputSchema": {
 "type": "object",
 "properties": {}
 }
 }
 ]
}

That’s your hello command, seen from an agent’s side of the glass. The name is your tool’s name and the command path joined with an underscore; the description is the Short you gave it in part 1; the inputSchema is empty because hello has no flags yet. Add a flag and it shows up here as a property, with the type and help text you already wrote. There’s no second schema to keep in sync, because the command tree is the schema.

A few things are deliberately left off this list: hidden and deprecated commands, pure command groups that don’t do anything themselves, and the mcp, help and completion plumbing. So mcp tools doubles as an audit: it’s exactly what an agent can reach, and nothing else.

Run the server

One command turns the whole thing on:

./bin/mytool mcp start

It doesn’t print a banner and march off doing things. It sits quietly, speaking MCP as JSON-RPC over standard input and output, waiting for an assistant to talk to it. You won’t run this by hand much; the assistant launches it for you. But it’s worth knowing what happens when the agent calls one of your tools: the server re-runs your own binary with the arguments the agent supplied, captures the output, and hands it back. The agent isn’t poking at your internals. It’s running mytool hello, the same command a human would, and getting the same result.

Point an assistant at it

The quickest way is to let the tool write the client config for you. For Claude Desktop:

./bin/mytool mcp claude enable

There are cursor and vscode variants too. Restart the assistant and your CLI is in its toolbox.

If you’d rather wire it by hand (or your client isn’t one of those three), the config is small. Point the client at your binary with mcp start as its arguments, using the absolute path:

{
 "mcpServers": {
 "mytool": {
 "command": "/absolute/path/to/bin/mytool",
 "args": ["mcp", "start"]
 }
 }
}

Claude Desktop keeps that in claude_desktop_config.json (under ~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows); Cursor uses ~/.cursor/mcp.json; VS Code’s Copilot reads github.copilot.mcpServers in your settings. The shape is the same everywhere. From here, ask the assistant to say hello and watch it call mytool_hello.

The agent’s reach is exactly your CLI’s reach

This is the part worth being calm about. Exposing your CLI over MCP doesn’t widen its surface by an inch. The agent can call the commands you shipped, with the parameters you defined, and nothing else. It can’t invent a command or pass a flag you never wrote. The boundary of what it can do is the boundary you drew when you built the tool, and mcp tools shows you that boundary in full. If there’s a command you don’t want an agent touching, mark it hidden and it drops off the list.

For a long-running or remote setup there’s also ./bin/mytool mcp stream, which serves the same tools over HTTP instead of stdio; the mcp reference has the details. For most desktop assistants, start over stdio is all you need.

What it comes down to

You turned the CLI you’ve been building into an agent-callable tool with one command and zero lines of AI code, because the real work, naming your operations and documenting their inputs, you finished the moment your --help was any good. Every command you add from here is a new tool the agent gets for free.

Next part goes the other way: instead of letting an assistant drive your tool from outside, we put AI inside it, wiring up a provider and building a feature against go-tool-base’s chat SDK. Until then, add a command or two and watch them appear in mcp tools. The agent’s toolbox grows as your CLI does.

Building a CLI with go-tool-base, part 2: configuration your tool can trust

Thu, 21 May 2026 00:00:00 +0000

In part 1 you scaffolded a tool and gave it a hello command. It says the same thing every time, which is fine for a first command and useless for a real one. The moment a tool does anything worth doing it needs settings: an endpoint, a default, a token, a log level. And the moment you have settings, you have the problem nobody warns you about. You set one in a file, the tool ignores it, the code that reads it looks perfectly correct, and an hour later you find you’d typed tiemout. Nothing in the whole stack thought that worth a word.

The good news is you don’t have to build any of this. Your scaffold already wired up a config system in part 1, the same one the rest of go-tool-base uses. This part puts it to work: where a setting’s value actually comes from, how to ship sensible defaults alongside the command they belong to, how to layer files so a team and a laptop can disagree politely, and how to turn a fat-fingered key from a silent shrug into an error that tells you exactly what you got wrong.

The same version note as part 1, since each of these stands on its own: everything here is written against go-tool-base v0.6.0 (gtb version will tell you what you’re on). The tool is young and still changing shape, so if you’re on a newer release and a detail has drifted, that’s the first thing to check. I’ll flag anything that breaks across versions as it comes up.

You already have a config system

The root command loads configuration for you before any of your command code runs, merges every source together, and hands the result to each command through Props. By the time your RunHello runs, props.Config is populated and ready.

A value can arrive from several places at once, so there’s an order. Highest wins:

Command-line flags
Environment variables (your tool’s prefix plus the key, so hello.greeting reads MYTOOL_HELLO_GREETING, with the dots turned into underscores)
Config files (on disk, in the order they were loaded)

That ladder is the mental model for what beats what: a flag beats an env var, an env var beats a file. The files are worth pinning down, though, because there’s more than one and they don’t all come from the same place. This is the bit that’s easy to trip over:

Embedded defaults are baked into the binary, one slice per command. You don’t read these at runtime directly. The init command (coming up) bakes them into your config file for you.
The file init writes, ~/.mytool/config.yaml, is the default the tool reads, along with a machine-wide /etc/mytool/config.yaml if one exists.
Files passed with --config replace those defaults for that run rather than adding to them. Name one or more and the tool reads exactly those.

We’ll set each of these up in turn. The full reference lives in the config docs.

Reading a value is one call, and it’s typed:

greeting := props.Config.GetString("hello.greeting")
timeout := props.Config.GetDuration("server.timeout")
debug := props.Config.GetBool("verbose")

Give a command a setting

Let’s make hello configurable. Open pkg/cmd/hello/main.go (your file, the one the generator leaves alone) and read the greeting from config instead of hard-coding it:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	greeting := props.Config.GetString("hello.greeting")
	props.Logger.Info(greeting)
	return nil
}

Build and run it:

just build
./bin/mytool hello

ERRO failed to load config: no configuration files found
please run init, or provide a config file using the --config flag

Not what you expected, maybe, but it’s the right instinct from the tool. It has no configuration to read yet, and rather than guess, it stops and says so. Which brings us neatly to where settings actually come from.

Defaults belong to the command

You could drop a default into the project’s central config, and for something truly global like the log level that’s the right home. But a setting that belongs to hello should live with hello, not in a shared file you have to remember to edit every time you add a command. The generator does this for you, you just have to ask. Back in part 1 you generated hello without config support, so run the same command again with --assets:

gtb generate command --name hello --short "Say hello" --assets

This is safe to re-run. The generator honours the code you’ve already written: it refreshes the boilerplate cmd.go, adds the asset scaffolding, and leaves your main.go, and the RunHello you’ve been editing, completely alone. One thing to hold off on here: don’t reach for --force. Force rewrites everything, including that main.go, which is exactly the work you want to keep.

You now have pkg/cmd/hello/assets/init/config.yaml, and the generator has already opened it under the command’s own namespace:

hello:

Fill in your defaults under it:

hello:
 greeting: Hello
 style: plain

Those values are embedded into the binary as an asset, and the generated cmd.go registers them with Props for you (props.Assets.Register("hello", &assets)), so the config system knows where your command’s defaults live. A quick word on style, since we’ll lean on it shortly: it’s a second setting I’m giving a default now so it’s ready when we need it. Plain says the greeting as written; loud will shout it.

That per-command home comes with one rule worth taking seriously: namespace your keys. Notice the generator opened the file under a hello: key rather than at the top level. Copy that. Every command ships its defaults in its own embedded file, and those files are all merged together to build the config, but the order they merge in is not guaranteed. If two commands both defined a top-level timeout, which one won would be a toss-up that could flip between builds. Keep each command’s settings under its own name (hello.greeting, report.timeout) and the clash can’t happen in the first place. The generator namespacing the file for you is a hint worth taking.

One thing the defaults file does not do is set values through struct tags. If you later add a default:"info" tag to a config field, that’s documentation for the error messages, nothing more. Real defaults live here, in the embedded YAML. It’s an easy thing to assume otherwise and then wonder why your default never applied.

First run: init

So your defaults are baked into the binary. The tool still needs an actual config file to read, and that’s what init is for. It’s one of the features your tool shipped with, so it’s already there:

./bin/mytool init

INFO Configuration initialised in /home/you/.mytool/config.yaml

Open that file and you’ll find your command’s defaults waiting in it, merged with the framework’s own:

hello:
 greeting: Hello
 style: plain
log:
 level: info

That’s the missing piece. init gathers every command’s embedded defaults through the Assets layer, writes them to ~/.mytool/config.yaml, locks the file down to 0600 (it may hold secrets later), and drops in a .gitignore so nobody commits it by accident. Now hello has something to read:

Prefer no init step? init is a feature, and you can leave it out of your tool’s feature set. With it off, the tool loads its embedded defaults directly and runs with no config file at all, you’d only add one to override something. That suits a small, self-contained tool. This tutorial keeps init on, which is the default and the right call while a tool is finding its feet, so the rest of the article assumes it.

./bin/mytool hello

INFO Hello

Setup that needs a human: initialisers

Static defaults cover the values you can decide for the user. Some you can’t: a token, an API key, an endpoint that differs per person. Writing a blank or guessed value for those is worse than useless. This is where go-tool-base does something I’ve not seen many CLI frameworks bother with: it lets a command bring its own first-run setup, and wires it in for you. It’s one of the genuine reasons to build on the framework rather than roll your own, so it’s worth a proper look.

Generate a command with --with-initializer:

gtb generate command --name greet --short "Greet someone" --with-initializer

Alongside the usual files you get an init.go. It’s generated and marked DO NOT EDIT, and it does all the wiring. Here’s the heart of it:

// Code generated by gtb. DO NOT EDIT.
package greet

func init() {
	setup.Register(props.FeatureCmd("greet"),
		[]setup.InitialiserProvider{func(p *props.Props) setup.Initialiser {
			if skipGreet {
				return nil
			}
			return &GreetInitialiser{}
		}},
		[]setup.SubcommandProvider{func(p *props.Props) []*cobra.Command {
			return []*cobra.Command{NewCmdInitGreet(p)}
		}},
		[]setup.FeatureFlag{func(cmd *cobra.Command) {
			cmd.Flags().BoolVar(&skipGreet, "skip-greet", false, "skip initializing greet configuration")
		}},
	)
}

type GreetInitialiser struct{}

func (i *GreetInitialiser) Name() string { return "greet" }

func (i *GreetInitialiser) IsConfigured(cfg config.Containable) bool {
	return cfg.IsSet("greet")
}

func (i *GreetInitialiser) Configure(p *props.Props, cfg config.Containable) error {
	return InitGreet(p, cfg)
}

That package init() registers three things with the framework the moment your command is imported, with no central setup file for you to edit: the initialiser itself, an init greet subcommand so the user can reconfigure just this command later, and a --skip-greet flag on the main init. IsConfigured is how the framework avoids nagging: if the greet key is already in the config, init leaves it be and moves on.

All of that is generated for you. The one piece that’s yours is the InitGreet function in main.go, which starts as a stub:

func InitGreet(p *props.Props, cfg config.Containable) error {
	// TODO: Implement custom initialization logic for greet
	return nil
}

Fill it in with whatever the setup needs. go-tool-base leans on huh for prompts, the same library its own GitHub and AI setup use, so a one-question form looks like this:

func InitGreet(p *props.Props, cfg config.Containable) error {
	var greeting string

	form := huh.NewForm(
		huh.NewGroup(
			huh.NewInput().
				Title("What greeting should greet use?").
				Value(&greeting),
		),
	)
	if err := form.Run(); err != nil {
		return err
	}

	cfg.Set("greet.greeting", greeting)
	return nil
}

Set the value on cfg and you’re done. After the initialisers run, init writes the whole config out to disk, so the answer persists into ~/.mytool/config.yaml with everything else. Run mytool init on a fresh machine now and it stops to ask for the greeting; run it again and it sails past, because IsConfigured sees the key is already there. Need to redo just this one command’s setup? mytool init greet. The framework hands each command its own setup step, its own subcommand and its own skip flag, and asks you for a single function in return. That’s the trade worth making: static defaults in your embedded YAML, anything that needs a human in an initialiser.

Overriding: the environment and layered files

With a config file in place, the other sources come into their own. The quickest override is an environment variable. Remember the prefix you set when scaffolding in part 1: hello.greeting maps to MYTOOL_HELLO_GREETING, the prefix and key joined up, uppercased, dots turned to underscores:

MYTOOL_HELLO_GREETING="Hello from mytool" ./bin/mytool hello

INFO Hello from mytool

You didn’t register that variable anywhere; the config system binds it for you. The prefix is what keeps it from colliding with some other tool’s LOG_LEVEL on the same machine, which is exactly why it’s worth having.

Files are the other half, and they’re where that precedence list earns a closer look. A single config file is fine until two people, or two machines, want slightly different settings, and then you’re copying files around by hand. The --config flag fixes that: pass it more than once and the tool merges the files in order.

./bin/mytool hello \
 --config ./config.yaml \
 --config ./config.local.yaml

Between the files you name, the rule is later wins on a clash, and every key that doesn’t clash is kept. If config.yaml sets hello.greeting: Hello and config.local.yaml sets hello.greeting: Oi, you get Oi, but keys that appear in only one file survive untouched. It’s a merge between them, not a replacement.

The edge to remember is what --config does to the default locations: it replaces them. The moment you name a file, ~/.mytool/config.yaml drops out of the picture unless you name it too. So you pass the whole stack you want, a shared base and a local override together, and let precedence settle it. Commit a config.yaml with the team’s settings, keep an untracked config.local.yaml for your own, run with both, and your local tweaks win without anyone editing a shared file. Leave --config off and you’re back on the defaults init wrote: ~/.mytool/config.yaml plus that machine-wide /etc/mytool/config.yaml if it’s there. Whichever set of files you land on, environment variables and flags still sit on top.

The typo that does nothing

Now for the failure I keep circling. Say you want to change the greeting. Open your config, but fat-finger the key:

hello:
 greting: Oi  # meant to be greeting

Run it, and you get a blank line. The greeting you set never applied: the misspelled key was read, matched nothing, and was silently dropped, and the real greeting is now nowhere to be found. Nothing said a word. For a greeting it’s a shrug. For a timeout or a retry count it’s the bug you chase at 2am, and I wrote up the why of it in the config key that quietly did nothing.

go-tool-base won’t catch this for you by default, and that’s a choice rather than an oversight. There’s no central schema that knows every key your tool could ever take, because keys belong to the commands that use them. What you get instead is a way to opt a command in, so it validates its own slice and nobody else’s.

Making mistakes loud

Tell the generator you want validation for a command and it scaffolds exactly this (gtb generate command --name hello --with-config-validation). Since hello already exists, it’s a small file to add by hand. Create pkg/cmd/hello/config.go:

package hello

import "gitlab.com/phpboyscout/go-tool-base/pkg/config"

// HelloConfig describes the config keys the hello command consumes.
type HelloConfig struct {
	Greeting string `config:"hello.greeting" validate:"required"`
	Style string `config:"hello.style" enum:"plain,loud" default:"plain"`
}

// ValidateHelloConfig checks the hello config against its schema.
func ValidateHelloConfig(cfg config.Containable) error {
	return config.ValidateStruct[HelloConfig](cfg)
}

The tags carry the rules. validate:"required" means the key has to be present and non-empty. enum:"plain,loud" means style has to be one of those two words. config.ValidateStruct[HelloConfig] does the rest: it derives a schema from those tags and checks the config against it, returning a readable error if anything is off. It takes props.Config as it is, the Containable interface, so there’s no casting to a concrete type. Call it at the top of RunHello, before you trust any of the values, and use the style while you’re there:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	if err := ValidateHelloConfig(props.Config); err != nil {
		return err
	}

	greeting := props.Config.GetString("hello.greeting")
	if props.Config.GetString("hello.style") == "loud" {
		greeting = strings.ToUpper(greeting)
	}

	props.Logger.Info(greeting)
	return nil
}

(You’ll add strings to the imports at the top of main.go.)

Now make a real mistake. Set the style to something that isn’t allowed:

hello:
 greeting: Hello
 style: shout

ERRO config validation failed:
 hello.style: value "shout" is not allowed (hint: allowed values: plain, loud)

That’s the difference. The command stops and tells you the key, the bad value, and what it would have accepted. The same check catches a misspelled greeting: the moment the real key goes missing, required fails with hello.greeting: required field is missing instead of quietly running on nothing. Set style: loud and you get HELLO, because the value finally passes and the code downstream can trust it.

If you switch on the optional config feature (it isn’t in the default set, so you opt into it), you also get a ready-made mytool config validate command that runs these checks without you wiring anything into a command at all. Either way, the principle holds: the program already knows what good config looks like, so make it say so when the config is bad.

The upshot

Your hello command now reads a real setting, ships a sensible default that init writes into place, honours overrides from the environment and from layered files in a predictable order, and refuses to run on a value it doesn’t understand. That’s most of what configuration ever needs to be, and you wrote almost none of the machinery.

One thing I’ve skipped: config can also reload while the tool is running, so a long-lived process picks up a changed file without a restart. That’s its own capability with its own moving parts, and I pulled it apart in reloading config without a restart if you need it.

Next part, we give the tool something to do with all this config: we turn it into an AI tool, with a chat command and an MCP server. Until then, go add a couple of validated settings to your own commands. You’ve got the shape of it now.

Building a CLI with go-tool-base, part 1: scaffold and your first command

Wed, 20 May 2026 00:00:00 +0000

Every time I start a new Go CLI, the first hour goes the same way, and none of it is the actual tool. Config loading. A logger. An update command. An error path that prints something a human can act on. A help system. I built go-tool-base so I’d never write that hour again, and I’ve spent a good few posts explaining how the pieces work inside. This series is the other half: how you use it. By the end you’ll have a real CLI with all that wiring for free. This part scaffolds one and gives it its first command.

One note on shape before we start: each part stands on its own. Finish this one and you’ve got a working, buildable tool. Later parts add configuration, AI, self-update and telemetry, one at a time. Where you want to know how a piece works underneath, I’ll link the deep-dive as we go.

Install the gtb CLI

go-tool-base ships an automation CLI called gtb. Install it with the script from the installation docs:

curl -sSL \
 "https://gitlab.com/phpboyscout/go-tool-base/-/raw/main/install.sh" | bash

That fetches a pre-built release, embedded docs and all, and drops gtb in ~/.local/bin, so make sure that’s on your $PATH. Then check it’s there:

gtb version

One thing to get out of the way before we build: versions. As I write this, gtb version prints go-tool-base v0.6.0, and that’s what every command and snippet in this series is verified against. It’s a young tool that’s still moving quickly, and the install script always pulls the latest release, so if you’re reading this later and something doesn’t line up, a newer version is the likeliest reason. When a release changes something that matters to this series, I’ll cover it in a follow-up.

Scaffold a project

One command stands up a whole project, and gtb gives you two ways to drive it.

The direct way, with flags, is good for scripting and for repeating a setup exactly:

gtb generate project \
 --name mytool \
 --repo myorg/mytool \
 --description "My CLI tool" \
 --env-prefix MYTOOL \
 --path ./mytool

--env-prefix is worth setting now: it’s the prefix for the environment variables that can override your config later (so MYTOOL_LOG_LEVEL rather than a bare LOG_LEVEL that would clash with every other tool on the box). The wizard defaults it to your tool’s name in capitals; with flags it’s worth being explicit. We’ll lean on it in part 2.

(gtb generate cli is the same command, if you prefer that name.) Or leave the flags off and gtb walks you through an interactive prompt instead, which is the gentler way the first time:

gtb generate project

Either way, one of the choices is worth calling out now, because it explains something you’ll see in a minute: features. go-tool-base bundles a set of ready-made commands, self-update, embedded docs, a doctor health check, an MCP server, a changelog, OS-keychain storage, and you choose which ones your tool ships with at generation time, either through the --features flag or a checklist in the wizard. The default set is a sensible starting point, and you can add or drop features later. That is why, a moment from now, your brand-new tool already answers --help with commands you never wrote. The full flag list is in the generate reference.

What you just got

cd mytool and look around. It’s a complete, releasable project, not a hello-world:

mytool/
├── cmd/mytool/main.go # entry point
├── pkg/cmd/root/
│ ├── cmd.go # builds Props, wires the root command
│ └── assets/init/config.yaml # embedded default config
├── internal/version/version.go # version info, stamped at release
├── .gtb/manifest.yaml # the generator's record of your command tree
├── .github/workflows/ # lint, test, docs, release pipelines
├── justfile # build / test / lint / docs tasks
├── go.mod # with `go tool` deps pinned
└── ... # .golangci.yaml, .goreleaser.yaml, README, CHANGELOG

One file there is worth understanding before anything else: .gtb/manifest.yaml. It is the generator’s source of truth, a record of every command your tool has, how they nest, and a content hash of each generated file. You won’t edit it by hand, but gtb reads and rewrites it constantly. It is how the generator knows what your command tree looks like, and how it can tell whether you have changed a file it owns. Think of it as the map the generator builds from: it’s committed to git for you, and as long as it’s there, your tool’s structure stays reproducible. We’ll see it earn its keep when we regenerate.

The entry point, by contrast, is tiny, because the framework does the lifting. Here’s the generated cmd/mytool/main.go in full:

// Code generated by gtb. DO NOT EDIT.

package main

import (
	"mytool/internal/version"

	gtbRoot "gitlab.com/phpboyscout/go-tool-base/pkg/cmd/root"
	"mytool/pkg/cmd/root"
)

func main() {
	rootCmd, p := root.NewCmdRoot(version.Get())
	gtbRoot.Execute(rootCmd, p)
}

Two lines of body. root.NewCmdRoot (in your pkg/cmd/root/cmd.go) builds a Props, the container that carries the logger, config, filesystem and version to every command. gtbRoot.Execute runs it and routes any failure through one consistent error handler, so there’s no os.Exit scattered about. Note the DO NOT EDIT header: main.go and the root cmd.go belong to the generator. Your code goes elsewhere, which matters in a minute.

Build it and you already have a working CLI:

just build # or: go build -o bin/mytool ./cmd/mytool
./bin/mytool --help

You’ll see the built-in commands from the features you picked, update, docs, doctor and the rest, with not a line written by you.

There’s one step before those commands will actually run. Try one, say ./bin/mytool docs, and the tool stops with please run init: it has no configuration yet and won’t guess at one. So give it some:

./bin/mytool init

That writes ~/.mytool/config.yaml from the defaults your tool ships with, and now its commands run. (init is itself one of the features. You can switch it off for a tool that should run straight from its built-in defaults with no file at all, but leave it on for now.) Part 2 takes configuration apart properly; for now, init once and carry on.

Add your first command

Don’t hand-roll a command file. gtb generates the boilerplate and leaves you the logic:

gtb generate command --name hello --short "Say hello"

That creates two files (see the command reference):

pkg/cmd/hello/cmd.go (generated, DO NOT EDIT): the options struct, flag wiring, and the NewCmdHello(props *props.Props) constructor.
pkg/cmd/hello/main.go (yours): a RunHello function, where all your real business logic goes.

The split is the whole point. Open pkg/cmd/hello/main.go and write what the command does:

func RunHello(ctx context.Context, props *props.Props, opts *HelloOptions, args []string) error {
	props.Logger.Info("hello from mytool")
	return nil
}

Rebuild, and the command is wired into the tree:

just build
./bin/mytool hello

You never touched the root command to register it. gtb recorded hello in that .gtb/manifest.yaml and wired it in for you. (If you’d rather wire commands by hand against the library directly, the custom-commands how-to shows that path; the generated route is the one this series follows.)

Regenerate without losing your work

Here’s the bit people are right to be wary of. If the generator owns cmd.go and the root wiring, what happens when it runs again, after you’ve made changes? And it runs often: every gtb generate command rebuilds the wiring.

gtb regenerate project

Your edits survive, and not by luck. Three separate things protect them:

Your logic sits in a file the generator never rewrites. Command logic lives in main.go; only the boilerplate cmd.go is regenerated. The split isn’t cosmetic, it’s the contract.
It notices if you edited a generated file. That manifest stores a content hash of every generated file, so if you’ve changed one, regeneration stops and asks before overwriting rather than silently stamping over you.
You can fence files off entirely. A gitignore-style .gtb/ignore tells the generator to leave specific paths alone, even under --force.

I wrote up how that edit-preserving diff actually works if you want the mechanism; the regenerate reference has the flags. For now, the thing to trust: scaffolding here is not a one-way door. You keep regenerating as the tool grows, and your edits stay put.

Where this leaves you

A few minutes in, you have a real CLI: config, logging, a consistent error path, self-update, embedded docs and a release pipeline, none of it written by you, plus your own hello command and the confidence to regenerate without fear. That’s the head start go-tool-base exists to give.

Next part: configuration. Typed settings, defaults the tool ships with, and how to turn a misspelled config key from a silent shrug into an error that tells you what you got wrong rather than a mystery you debug at 2am. Until then, go add a few more commands. You’ve got the pattern now.

Migrating away from Mediawiki and how to export its data

Wed, 19 Aug 2020 00:00:00 +0000

I like Mediawiki, it is a simple tool capable of doing a lot and can be very flexible and easy to customise. However its not always the right solution! I had a situation where we needed to migrate away from using it for a combination of security and usability reasons. So I thought it would be good to document it.

After reviewing a few things it was decided to move things over to the companies already existing O365 SharePoint as a new site. This sounded laborious as first, but actually turned out to be pretty straight forward.

We start with getting data out of Mediawiki, thankfully we only wanted the most recent revision and not the full history of a page. We use PostgreSQL as a back-end so it was reasonably straight forward to figure out how to extract the data in a sensible query.

SELECT 
 page_id as id, 
 page.page_title as title, 
 pagecontent.old_text as content, 
 page_touched as edited
FROM mediawiki.page
LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
ORDER BY page_touched DESC;

It tool a little sleuthing to realize that the slots table was the pivotal in extracting the latest page version. With the right join and a little mangling of the content_address field from the contents table to remove the “tt:” from the value and convert to an integer we now have a result set of all the page names and the latest revision of that page. I also added in the date the page was last updated to allow me to see when it was last edited as it was a live system migration and helped me to ensure content remained sync while both were still in play.

Once I had the query it was a simple job of writing an “Exporter” using Go Lang to extract the data and write it to files, I’ll chuck a snippet of code at the bottom of the post for you.

Mediawiki uses wikitext as a format so I needed to convert it to something more widely understood. Having used Pandoc in the past successfully I plumped for this as I knew it would handle a lot of options and was simple to use to convert to the markdown_mmd format

I Installed it via the ubuntu apt package available on my system and wired this in as a hacky exec command into my script… and voila! I had hardcopies of all the Mediawiki pages on my system in both wikitext and markdown_mmd format.

Why markdown_mmd I hear you ask… mainly because it gave me the cleanest conversion for use with the new markdown web page widget for Sharepoint’s modern interface.

Now we have the files we could do a little munging and parsing to convert URLs into the format needed for the new location in Sharepoint, easily done with a bit of regex pattern matching, which I wont go into as yours will be very different from mine… suffice to say looking for "wikilink" in my regex helped massively in finding all the occurrences I needed to update. I used sed but you could use whatever tool you like or add it into your version of the exporter

'SysAdmin/(.+) "wikilink"'

and with a little back referencing to substitute the values we need to keep and its all good.

Next came the import of the data into Sharepoint, but that is a post for another day.

package data

import (
	"bytes"
	"fmt"
	"github.com/jmoiron/sqlx"
	"github.com/rs/zerolog/log"
	"io/ioutil"
	"os"
	"os/exec"
	"path/filepath"
	"time"
	"wiki-export/src/util"
)

type Page struct {
	Id int
	Title string
	Content string
	Edited time.Time
}

type Exporter struct {
	Config util.ExporterConfig
	DB *sqlx.DB
}

func (l *Exporter) Export() {
	stmt := `
		SELECT page_id as id, page.page_title as title, pagecontent.old_text as content, page_touched as edited
		FROM mediawiki.page
		LEFT JOIN mediawiki.slots ON page.page_latest = slots.slot_revision_id
		LEFT JOIN mediawiki.content ON content.content_id = slots.slot_content_id
		LEFT JOIN mediawiki.pagecontent ON pagecontent.old_id = CAST(OVERLAY(content.content_address placing '' from 1 for 3) as integer)
		ORDER BY page_touched DESC
		;`

	page := Page{}
	rows, err := l.DB.Queryx(stmt)
	util.CheckErr(err)

	for rows.Next() {
		util.CheckErr(rows.StructScan(&page))
		wikiFilename := fmt.Sprintf("%s.mediawiki",filepath.Base(page.Title))
		mdFilename := fmt.Sprintf("%s.md",filepath.Base(page.Title))
		path := filepath.Dir(page.Title)
		wikiDir := fmt.Sprintf("%s/mediawiki",l.Config.TargetDir)
		mdDir := fmt.Sprintf("%s/%s",l.Config.TargetDir, l.Config.TargetFormat)

		if path != "." {
			wikiDir = fmt.Sprintf("%s/mediawiki/%s",l.Config.TargetDir , path)
			mdDir = fmt.Sprintf("%s/md/%s",l.Config.TargetDir , path)
		}
		util.CheckErr(os.MkdirAll(wikiDir, 0777))
		util.CheckErr(os.MkdirAll(mdDir, 0777))

		wikiTarget := fmt.Sprintf("%s/%s", wikiDir, wikiFilename)
		mdTarget := fmt.Sprintf("%s/%s", mdDir, mdFilename)

		log.Debug().Msgf("%s => %s -> %s", page.Title, wikiTarget, mdTarget)

		c := []byte(page.Content)
		util.CheckErr(ioutil.WriteFile(wikiTarget, c, 0777))

		cmd := exec.Command("pandoc", "-f","mediawiki", "-t", l.Config.TargetFormat, wikiTarget)
		
		var errorBuffer bytes.Buffer
		var outputBuffer bytes.Buffer
		cmd.Stdout = &outputBuffer
		cmd.Stderr = &errorBuffer
		err := cmd.Run()
		if err != nil {
			log.Err(err).Msgf("ERROR: %s", errorBuffer.String())
			util.CheckErr(err)
		}

		util.CheckErr(ioutil.WriteFile(mdTarget, outputBuffer.Bytes(), 0777))
	}
}

Adding Ambient Sounds to your Discord Server On Linux

Tue, 30 Jun 2020 00:00:00 +0000

I’m a Dungeon Master! I don’t mean that in the S&M sense! As in the game Dungeons & Dragons (https://dnd.wizards.com), where I run a weekly game as well as take part in a couple of campaigns as a player. It’s a lot of fun and something I would definitely recommend you have a go at if you are so inclined

There is a vast amount of tooling & tech out there that allows you to play remotely such as Virtual Table Tops, Character builders, online resources, etc. One such tool that gets used quite often is a chat service called Discord (https://discord.com) It’s really useful and allows you to easily be part of and manage communities of people…. Think IRC & Slack, but more up to date than IRC and less “workish” than Slack.

As part of my online games I like being able to have ambient music to match the surroundings the players are traveling through, as well as some active elements thrown in for good measure. This is possible in a few different ways using discord but the way I want to set it up can be somewhat frustrating to set up. Let me explain:

I have taken a shine to two tools in particular… Syrinscape (https://syrinscape.com) and Table Top Audio (https://tabletopaudio.com). The former being a windows app with an nice interactive mixing UI that allows you to combine and generate unique sounds, the latter being a lovely web service that has some fantastic loop-able ambient background tracks all 100% free.

I am wanting to be able to pipe the audio from these two services into my Discord server so that I can make use of the fantastic audio they offer. This is the journey of how I managed to get this working, partly as a reminder for me if I ever need to do this again and also to help others that may be looking to do the same.

My Setup

I’ve been a big fan of Ubuntu for a number of years, but since 20.04 I’ve found that the shine I’ve had for it has waned significantly. I wont go into the why and wherefore of it but I’m now running the excellent Pop_OS! from System76 (https://pop.system76.com) its an Ubuntu variant but with the bits I dislike removed. So assume that anything I’m doing is compatible with Ubuntu 20.04.

The Requirements

The ideal solution should see me being able to have a single instance of discord running that allows me to still use my mic to be able to talk, and to have my selected background playing with the ability to control the volumes of both the mic and the background independently.

Finding A Solution

A lot of googling led me to realise that there isn’t a perfect solution to fit my brief. The hardest part being not actually knowing what to google and a lot of the terminology being somewhat foreign to me as I’m not much of an audio engineer. However I finally stumbled upon a blog post by Emma Anderson dated June 2016 and thankfully it gives me a lot of the heavy lifting that I needed along with some explanation of what I’m trying to achieve, though I’m hopefully going to be more verbose here in what this all means and how it works.

PulseAudio

The first thing we need to do is make sure the packages for pulseaudio and pavucontrol are installed. These will allow us to manipulate the way we capture sound and redirect it to the appropriate inputs and outputs.

apt install pulseaudio pavucontrol

Virtual Input & Virtual Mic

What we are going to try to achieve, is to create two new elements inside of Pulseaudio;

a Virtual input that we can channel the applications creating our background sounds which will allow us to control the volume independently.
a Virtual Microphone that we can channel our both our normal microphone and the new Virtual input into.

By creating these elements we can then use the pavucontrol tool to select what needs to be redirected where. so lets get started.

pactl load-module module-null-sink sink_name=VirtualInput
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

Here we have two commands, the first will create our new Virtual Input as what is referred to as a “null sink”. This on its own is not really very useful for us as we also need what is referred to as a “source”, thankfully when we run this command it also created a new “source” for us.

On it’s own that should be more than enough, but running the 2nd & 3rd command makes our live a lot easier because it will apply some very useful labels to both of the newly created sink and source. In this case VirtualInput for the sink and VirtualInput.monitor for the source. Having these in place makes it a lot simpler to configure things with pavucontrol.

Next we need to create our Virtual Mic using some very familiar looking commands.

VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

again we have now created a new new pair of sink and source with some nice easy to recognise labels that we will use when we start working with pavucontrol.

The next piece to our puzzle is creating the components that will let us define a connection from the VirtualInput and our physical microphone to the newly created VirtualMic. We do this with two identical commands;

pactl load-module module-loopback sink=VirtualMic
pactl load-module module-loopback sink=VirtualMic

we now have most of the elements that we need to configure everything to work.

Listening to my own Ambience

Before we can start wiring it all together we need to ensure we can also listen back to our own ambience. This involves us creating one more “loopback” module that points to the speakers we are wanting to listen to. Lets find out what our options are by running;

pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device.description/ {print $0}

This lists all of the available “sinks” that we can use. on my daily driver laptop I get;

 * index: 1
	name: <alsa_output.pci-0000_00_1f.3.analog-stereo>
		device.description = "Built-in Audio Analogue Stereo"

This tells us the “index” for the device, its name and also some kind of description. The important bit for us here is the name as we will need that to create our new “loopback” with the command;

pactl load-module module-loopback sink=alsa_output.pci-0000_00_1f.3.analog-stereo

This creates the last piece for our puzzle!

Connecting it all together

I’m now going to assume you have logged yourself into the Discord client and fired up your copy of Syrinscape… but you should just as easily swap out these for something else of your choice.

Now we can start pavucontrol either from the command line or you can look for it in your applications menu. Once it loads you will hopefully be presented with something that looks like;

For this next step I am specifically starting on the “Recording” tab of pavucontrol this is to allow us to set up what is going to be captured. I have updated the drop-down at the bottom left to show “All Streams” as this will make it quicker to configure… Starting at the top we have two entries for;

Loopback to VirtualMic from: These are the result of the first two “loopback” modules we created with the pactl command we ran previously. They are going to allow us to capture the audio streams from our physical microphone, mine here is the TONOR TC-777 and our newly created VirtualMic.

followed by a single entry for;

Loopback to Built-in-Audio Analougue Stereo from: which is the last “loopback” module that we create to let us hear our own Ambience, Having this set to our VirtualInput means that anything that pipe into our VirtualInput will also come out of our speakers.

and finally;

WEBRTC VoiceEngine: Once you connect to a voice channel in discord this will appear and it allows us to specify which of our devices it should be reading the audio feed from. For our purposes we have this set to our VirtualMic so that we can have our mixed audio feeds

Now that recording is configured we can sort out our playback.

Here we can see the “Playback” tab of pavucontrol, again set to show “All Streams”. This time I’m going to run through the elements here starting from the bottom of the list and working my way up…

WEBRTC VoiceEngine: This again is our connection to a Discord voice channel, as you can see I have this set to play back all of its output via Built-in Audio Analogue Stereo which is how my Operating system has labelled my physical speakers.

Syrinscape.exe: This is the Syrinscape application, that I run through PlayOnLinux (https://www.playonlinux.com), and I will use to generate all of my lovely ambient sounds. This is set to play all of its “audio stream” on our VirtualInput.

The next two items in our list, providing you have configured the Recording tab first should look as in the image. Changing the “Loopback to VirtualMic” entries on the Recording tab will change the labels of these two entries.

Loopback of VirtualInput.monitor on: it seems we have two of these entries and where we can tell the to pip all of the audio we are now capturing on our VirtualInput In this case we want it to go to two places, our VirtualMic so that it can be sent to both our Discord audio channel and also to our Built-in Audio Analogue Stereo speakers.

Loopback of Built-in Audio Analogue Stereo on: is where we now direct the input from our physical microphone and feed that straight into our VirtualMic.

The other entries in the list here are for firefox and the system itself and are not relevant to what we are trying to achive.

Winner Winner Chicken Dinner

That’s is effectively all we need to do… From here on in anything you play via the Syrinscape app will be merged with your microphone input and passed to Discord. You can then use the volume sliders in pavucontrol to adjust the levels of all the inputs to suit your own personal preference.

Though I will make a few small suggestions about how to configure your discord settings. You shouldn’t need to make any adjustments to the input and output devices which should now be set to Default as your “Input Device” if you change this it will override the changes we have made and you will need to go back to the Recording tab of pavucontrol and switch WEBRTC VoiceEngine back to InputMic , but…

I would recommend disabling automatic input sensitivity and lowering the sensitivity slider all the way down to -100dB… this is to allow for the potential low and subtle tones and ambient elements you may want to play… be warned though it makes it very very easy for an low quality microphone (such as the Built-in Audio Analogue Stereo microphone found on my laptop) to pick up other noises such as your systems fans, mouse clicks and typing. A simple way to combat this is to get a reasonable quality external cardioid condenser microphone which eliminates a lot of this unwanted background.

One last thing

That should be it for now… I’ll leave you with one final thing. This is a simple little bash script I threw together that I run in can a terminal to create all the components and if I want will then clean them all up and remove them. If you really want you could set it up as a permanent implementation, but I’ll let you google for that solution!

#!/bin/bash

LB1=""

listenback() {
 echo ""
 echo "Listing all possible output devices"
 pacmd list-sinks | awk '/index:/ {print $0}; /name:/ {print $0}; /device\.description/ {print $0}'
 echo ""
 echo "Please enter the name of the output device to create a loopback for (leave blank to skip): "
 read S1
 if [ "$S1" != "" ]; then
 	echo " * Creating Loopback for '$S1'"
 	LB1=$(pactl load-module module-loopback sink="$S1")
 fi
}

cleanup() {
 while true; do
 read -p "Finished? do you want to clean up and remove modules [Yn]: " yn
 case $yn in
 [Yy]* ) return 0;;
 [Nn]* ) return 1;;
 * ) echo "Please answer yes or no.";;
 esac
 done
}

listenback

echo " * Creating VirtualInput"
VI=$(pactl load-module module-null-sink sink_name=VirtualInput)
pacmd update-sink-proplist VirtualInput device.description=VirtualInput
pacmd update-source-proplist VirtualInput.monitor device.description=VirtualInput.monitor

echo " * Creating VirtualMic"
VM=$(pactl load-module module-null-sink sink_name=VirtualMic)
pacmd update-sink-proplist VirtualMic device.description=VirtualMic
pacmd update-source-proplist VirtualMic.monitor device.description=VirtualMic.monitor

echo " * Creating loopbacks for VirtualMic"
VML1=$(pactl load-module module-loopback sink=VirtualMic)
VML2=$(pactl load-module module-loopback sink=VirtualMic)


echo "All modules have been loaded have been configured! Run pavucontrol to configure your devices."

if cleanup; then
 pactl unload-module "$VML2"
 pactl unload-module "$VML1"
 pactl unload-module "$VM"
 pactl unload-module "$VI"
 if [ $LB1 != "" ]; then
 pactl unload-module "$LB1"
 fi
 echo "All modules have been unloaded"
else 
 if [ $LB1 != "" ]
 then
 echo "Modules $LB1, $VI, $VM, $VML1 & $VML2 remain loaded"
 else
 	echo "Modules $VI, $VM, $VML1, & $VML2 remain loaded"
 fi
fi

Encrypting additional drives with LUKS on Linux

Mon, 29 Jun 2020 00:00:00 +0000

Encryption is king nowadays with everyone having mobile devices. We have a significant number of people on laptops that travel around and also workstations that live in open plan offices. This means we encrypt all of our disks… just in case. 99% of the time is super simple to do as most OS installers give you the option to do it, some now ven enforce it as a default option. This post however is about adding an additional disk to the system and making it automatically mount on system startup.

So let me set the scene, we have a data-scientist that’s running out of disk space for a task they are running on their Ubuntu 18.04 Workstation. At some point the workstation had an upgrade to the HDD in the past to a shiny new SSD, and the old 4Tb spinning disk was left in the chassis that they want to use for this very specific task.

Now this workstation has been through a couple of data-scientists over the last 12 months and unfortunately the LUKS password that had been set up for the old spinning disk has gone walkabouts… so the plan is as follows

flatten the old disk and set up a new partition using the whole disk
generate a new secure encryption key
set up LUKS encryption on the new partition
use Ext4 as a filesystem
enable auto decryption of the disk
add the new partition to the fstab to mount on system startup

N.B Assume that we are running everything as the root user

Flatten the disk

As we cant recover anything we are going to flatten the disk using parted (apt install parted to install) to allow is to create a partition greater than 2Tb, but first we are going to identify the disk we are working with… I tend to favour using either fdisk -l or as a more concise option lsblk -p which gives us a an easy to interpret overview something like:

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk

I can tell from this that we are looking at using the disk that is currently at /dev/sdb and its showing as being 3.7Tb in size.

Great… now to set up our new partition using the command parted /dev/sdb which gives us an interactive shell to work with (you can see the prompts in the output below are prefixed with (parted)

GNU Parted 3.2 
Using /dev/sdb 
Welcome to GNU Parted! Type 'help' to view a list of commands. 
(parted) mklabel gpt

The command mklabel gpt will wipe the partition table for /dev/sdb and give us a clean slate to work from

(parted) unit TB

We now set parted to think in Terabytes as the default reference size using the command above.

(parted) mkpart primary 0.00TB 3.70TB

Now we get to create the actual partition. You can see from the command above that we are using the command mkpart and telling it to create a primary partition type.

(parted) print 
Model: ATA WDC WD4005FZBX-0 (scsi) 
Disk /dev/sdb: 4.00TB 
Sector size (logical/physical): 512B/4096B 
Partition Table: gpt 
Disk Flags: 
 
Number Start End Size File system Name Flags 
 1 0.00TB 4.00TB 4.00TB primary

We can chek everything went smoothly using the print command which gives us confirmation that a new primary partition is present. We can now leave parted with a simple.

(parted) quit

And we can now use fdisk -l or lsblk -p to see that we now have a partition waiting for us at /dev/sdb1.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part

Generating an encryption key

Our disk is now ready for use, but not yet encrypted, so our next step is to create a key that can be used when we encrypt the disk. As we are going to be mounting it automatically we want to use a keyfile to store the key. You can of course create a key by mashing the keys on the keyboard, but I tend to prefer letting something else do the hard part for me.

First we create somewhere to store the key… I opted for,

mkdir -p /etc/crypt/keys

But feel free to put it wherever you want just as long as its only accessible by the root user. Next we generate the keyfile using the command:

dd bs=512 count=4 if=/dev/urandom of=/etc/crypt/keys/sdb1 iflag=fullblock

Here I am using /dev/urandom as my randomness generator, but you could use any valid generator of your choice. With this set of parameted dd will read the stream of “randomeness” and write 2048 bytes to our keyfile at /etc/crypt/keys/sdb1. If you want to be a little more complex about teh size and shape of your key then have a look at https://man7.org/linux/man-pages/man1/dd.1.html

Encrypting the Disk

Hopefully it will already be installed because you encrypted your root disk at installation, but if not you can run apt install cryptsetup to get going.

The command to do the encryption is actually very simple.

cryptsetup luksFormat /dev/sdb1 /etc/crypt/keys/sdb1

You can see that using the cryptsetup tool we are asking it to execute teh command luksFormat but while it says format in the command this is a little misleading as it doesn’t actually format the disk but just rewrites a portion of bytes at the beginning of the partition to enable encryption. we then tell it the partition we want encrypting, here its /dev/sdb1 and finally we pass in the keyfile we just generated and saved at /etc/crypt/keys/sdb1. If you omit the keyfile it will still encrypt teh disk but will prompt you to enter the key manually.

As soon as you press enter you will be warned of teh danager of what you are doing… so double check you are encrypting the right partition and follow the instructions that should look something like :

WARNING! 
======== 
This will overwrite data on /dev/sdb1 irrevocably. 
 
Are you sure? (Type uppercase yes): YES 
Command successful.

And that’s it… the disk is encrypted and ready to use. There are a few ways you can now work with the disk. the quickest and easiest is to just decrypt the disk manually using cryptsetup to open the disk.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

Here we open /dev/sdb1 and give is a new name of sdb1_crypt and we unlock it using the -d argument to tell it the keyfile we generated before.

That is the dis decrypted and ready to roll… you can now use fdisk -l or lsblk -p to confirm that it is now available at /dev/mapper/sdb1_crypt.

/dev/sda 8:0 0 1.8T 0 disk 
├─/dev/sda1 8:1 0 512M 0 part /boot/efi
├─/dev/sda2 8:2 0 732M 0 part /boot
└─/dev/sda3 8:3 0 1.8T 0 part 
 └─/dev/mapper/sda3_crypt 253:0 0 1.8T 0 crypt 
 ├─/dev/mapper/ubuntu--vg-root 253:1 0 1.8T 0 lvm /
 └─/dev/mapper/ubuntu--vg-swap_1 253:2 0 976M 0 lvm [SWAP]
/dev/sdb 8:16 0 3.7T 0 disk 
└─/dev/sdb1 8:17 0 3.7T 0 part 
 └─/dev/mapper/sdb1_crypt 253:3 0 3.7T 0 crypt /mnt/4tb-1

This tells us that the newly decrypted disk is now available at /dev/mapper/sdb1_crypt and is a volume of 3.7Tb… Exactly what we were hoping for!

All finished with your encrypted disk… you can just as easily close it again using:

cryptsetup close sdb1_crypt

Setting up the Filesystem

Ok, we have an encrypted partition, we can decrypt it but we cant mount it yet as we don’t have a file system to work with. Let’s take care of that real quick by opening up the partition again.

cryptsetup open /dev/sdb1 sdb1_crypt -d /etc/crypt/keys/sdb1

And now that its available we are going to set up an ext4 filesystem using the command mkfs.ext4 /dev/mapper/sdb1_crypt which, all going according to plan, should look something like:

mke2fs 1.44.1 (24-Mar-2018) 
Creating filesystem with 976753664 4k blocks and 244195328 inodes 
Filesystem UUID: d797be67-c53e-49d3-897e-c624b21a22d3 
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
 102400000, 214990848, 512000000, 550731776, 644972544 
 
Allocating group tables: done 
Writing inode tables: done 
Creating journal (262144 blocks): done 
Writing superblocks and filesystem accounting information: done

we are now good to go… lets try mounting the filesystem with

mount -t ext4 /dev/mapper/sdb1_crypt /mnt

I’m just mounting straight to /mnt but obviously this can be any folder you want. If the command worked we can easily confirm it with a quick df -h:

Filesystem 1K-blocks Used Available Use% Mounted on
udev 32846708 0 32846708 0% /dev
tmpfs 6578140 2308 6575832 1% /run
/dev/mapper/ubuntu--vg-root 1919562064 993193376 828790500 55% /
tmpfs 32890688 200 32890488 1% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 32890688 0 32890688 0% /sys/fs/cgroup
/dev/sda2 721392 276068 392860 42% /boot
/dev/sda1 523248 6232 517016 2% /boot/efi
/dev/mapper/sdb1_crypt 3844637680 0 3844637680 1% /mnt

Excellent… you can now start working with your new partition… however lets un-mount and close the drive quickly with a

umount /mnt && cryptsetup close sdb1_crypt

And then we can move onto…

Automatic Decryption

This is a lot simpler that you may realise… all we need to do is add a new line to the file /etc/crypttab! But first we need one last piece of information we don’t yet have, but we can easily get with the command

sudo cryptsetup luksDump /dev/sdb1 | grep "UUID"

This will use luksDump to get information about the encrypted partition and then uses grep to specifically target the property UUID which we will need to identify the partition in the next step.

Now in your favourite editor of choice add the following line, replacing the spoof UUID here with the one we just found.

sdb1_crypt UUID=1111111111-2222-3333-4444-555555555555 /etc/crypt/keys/sdb1 luks

Here we are giving the decrypted volume a unique label for teh decrypted label to be made available at the appropriate /dev/mapper/* location. We also specify the UUID to identify the partition to decrypt… we could use the path /dev/sdb1 but using the UUID is more explicit and prevents any confusion if another partition happens to present itself as /dev/sdb1 at some point in the future. Third we have the path to our newly generated keyfile and finally we have the encryption mode that we are using for encryption which here is luks. For more info on crypttab have a look at https://www.freedesktop.org/software/systemd/man/crypttab.html

We can now test that auto decryption is working using:

cryptdisks_start sdb1_crypt

which if successful should have an output like:

 * Starting crypto disk... 
 * sdb1_crypt: INSECURE MODE FOR /etc/crypt/keys/sdb1, see /usr/share/doc/cryptsetup/README.Debian. 
 * sdb1_crypt (starting).. 
 * sdb1_crypt (started)...

All that’s left to do now is set up

Auto-mount the filesystem

Hopefully we now are on really familiar ground… we can now treat /dev/mapper/sdb1_crypt as a bog standard ext4 partition that can be mounted via the /etc/fstab by adding the line:

/dev/mapper/sdb1_crypt /mnt ext4 defaults 0 2

As you can see its pretty ordinary, exactly as you would expect, obviously swapping out /mnt with the location of your choice to mount the filesystem. If you are not wholly familiar with fstab then its definitely worth having a look at https://help.ubuntu.com/community/Fstab as it gives a good overview for those who are new to it…

Finally we can check that it all works with:

mount -a

And at this point I am pretty sure I can hear the fat lady singing…

Connecting to Ubuntu 18.04+ using RDP

Mon, 20 May 2019 00:00:00 +0000

We have a mix of different setups that the Software Engineer and Data Scientists use to get their work done. There are some using just Linux on laptops, Some on MacBooks and some on the various versions of Windows.

For those not using Linux as their primary OS we have a bunch of Desktops that run Ubuntu 18.04+ for them to connect to. SSH can do quite a lot but a few of the team work remotely and in house we prefer RDP for that kind of thing rather than VNC.

We have had some issues with connections in the past so this post exists to remind me how next time I need to set it up. First we need to install the xRDP server package.

sudo apt install xrdp

Next we need to ensure that we have the right ports open on the workstation. If like me you also use UFW to manage your firewall rules then open port 3389 using…

sudo ufw allow 3389

The issue left, is that you will get an annoying pop up when you log in about a colour management profile needing to be set up and asking you to provide your password. Even then you may still get some annoying crash pop-ups.

I found a really good solution to this at http://c-nergy.be/blog/?p=12043 which I’ve cribbed and paraphrased below

Create the file /etc/polkit-1/localauthority/50-local.d/45-allow-colord.pkla (using your editor of choice and sudo)and add the following contents

[Allow Colord all Users]
Identity=unix-user:*
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
ResultAny=no
ResultInactive=no
ResultActive=yes

We now need to clear any crash dumps from the workstation

sudo rm /var/crash/*

You should then be good to connect to using whatever RDP client you prefer… I like Remmina myself but each to their own.

Dell DisplayLink D6000 & Ubuntu 18.04+ Issues

Tue, 14 May 2019 00:00:00 +0000

I love Ubuntu… I’m pretty fond of dell kit too!

So I was rather chuffed when I started working at Medicines Discovery Catapult because they let me have both. When you look at my desk it looks like it could be an advert for Dell. Laptop, monitors, dock, keyboard and mouse…. its great when you have a corporate account with a Dell reseller

However while I’ve had a lot of success with the D3000 DisplayLink dock on Ubuntu I found that I’m now having to deal with the upgraded D6000… which doesn’t play very nicely with the more recent versions of Ubuntu (we are talking 18.04 and later)

I kept finding that after a random amount of time the D6000 would randomly seem to power down… I would lose the screens, audio, networking and USB. and the only way I could fix it is to unplug it from teh laptop and plug it back in. Not ideal, especially if I’m in the middle of a video call or debugging something on the net

Being the kind of techie I am my first port of call checking my logs… but I couldn’t see anything that would cause this random disconnect. So off to google I went… eventually I found a lot of information telling me it was part of power management causing things to start powering down… In this case it implied that it was something trying to suspend USB… which sounded really plausible!

So a little more research suggested that I should be using laptop mode tools to disable the ability for USB to be suspended. I gave it a go, though I was dubious as in my mind I shouldn’t have needed to install an additional package (albeit a great one for tweaking your power management on a laptop running Linux)

Alas no joy! And I had too much to do to start debugging in depth and ripping apart other peoples code to figure it out.

What did I do? you ask. Well, I just put up with it for a few weeks, but gradually it began to grate on my nerves. However there was that one day where it didn’t turn off… and that left me perplexed… I checked if any updates had been applied in my last apt update && apt upgrade … nothing…. it then dawned on me that I had plugged in the headset I used for conference calling into the audio in/out on the dock instead of directly into the laptop.

Now I had a little more information I was able to deduce (with googles help) that the laptop was actually suspending USB, but that the trigger was actually pulseaudio. At this point it becomes really easy to solve the problem.

Solution

Edit /etc/pulse/default.pa using your preferred editor (and sudo)

Find the line

### Automatically suspend sinks/sources that become idle for too long
load-module module-suspend-on-idle

And comment it out and save!

Lastly, because its run as a user service you need to restart the Pulse Audio daemon using the command

systemctl --user restart pulseaudio.service

or you could just logout and back in again

Using Gmail aliases with Evolution

Wed, 06 Jan 2016 00:00:00 +0000

If your anything like me you have a large number of email aliases that you use with Gmail which is great. However I use Evolution as a mail client more often than not when using Gnome3 as a desktop.

It’s very easy to set up Evolution to create separate outbound email accounts that you can use for handling all of your aliases. It doesn’t yet support OAuth2 as an authentication mechanism for any account that is not set up using the built-in Gnome Online Accounts integration.

This is a real pain as Google have disabled the more common ‘plain’ and ’login’ authentication mechanisms for use with an SMTP only account. Meaning that any time that you try to connect to smtp.gmail.com:587 with STARTTLS you will get some form of error message to the effect of “Bad Authentication”.

Hopefully I’ll find a workaround at some point in the near future or Evolution will add the facility to enable OAuth2 as an available authentication mechanism.

In the mean time there is a workaround if you visit https://www.google.com/settings/security/lesssecureapps you can enable these less secure authentication mechanisms allowing you to once again connect and send email via email addresses using SMTP

A metaphor about PSR-7 and Middleware for non-developers

Thu, 08 Oct 2015 00:00:00 +0000

Never one to shy away from coming up with a metaphor for explaining something technical I found myself having to come up with one on the spot for PSR-7 and Middleware while at the recent PHPNW15 Conference.

Normally my brain will come up with something completely inappropriate but this time round I found I quite liked the imagery that came to mind.

If you would like to find out more of the specifics about PSR-7 you can take a look at http://www.php-fig.org/psr/psr-7/ which will make a better job of explaining it than I could ever do.

Now on to the metaphor

Imagine a house on fire, a bizarre way to start I know but bear with me. The nearest well with water that can put out the fire is 500 meters away! We then have a human chain stretching between the well and the house with a bucket going back and forth between trying to put the fire out. So lets break this down, the house represents the internet, or more specifically you and your browser. The fact you are on fire means that you are desperately needing water to quench the flames. At this point you send an empty bucket which represents your “request”, along the human chain, which in itself represents the application, to the well.

At the start of the chain the bucket is pretty normal, it’s a bucket of course, its round, made of wood with a rope handle, lets say it has a small leak in it.

As it travels down the chain it’s passed from person to person, everyone in it has the opportunity to do something with the bucket, or not as the case may be and could just pass it to the next person in the chain. Others may attempt to fix the leak in the bucket, someone may choose to replace it with a metal bucket, change the handle or make it bigger. Regardless of what may be done to the bucket in essence it remains a bucket.

Inexorably the bucket will continue to move down the chain to the well. When it reaches the well it changes state because now it has been filled with water. All of the interaction with the bucket thus far, mean that what happens at the well could vary depending on the changes have been made . If its been made bigger, for example, it could be filled with significantly more water, if swapped for a metal one it could imply that the bucket descends the well to get the water quicker because its heavier. Either way it is filled with water and begins its journey back towards the house.

Again it passes through the hands of each person in the chain, but now that its state has changed it now has the opportunity to be modified again. Someone may empty some water out as there is too much in the bucket, others may say that there is not enough and send it back down the line towards the well to be refilled. Either way the bucket continues to change hands over and over until it reaches the house and the contents are thrown on the fire to complete the request for water.

During this whole time the human chain could have been in flux. Some people may have swapped places, left the chain, added to the chain, some extraordinary people may have played leapfrog in the chain and appeared to handle the bucket more than once. Regardless of these changes the chain remains and continues to pass the bucket from one person to the another as long as the requests for water keep coming.

This, in the simplest possible form, explains PSR-7 and the concept of Middleware.

The bucket remains a bucket because PSR-7 says that is what is needed to complete the request for water, it also defines how you should interact with it regardless of what modifications have been made. If the bucket cant be used according to how PSR-7 describes a bucket to be, then the middleware can’t complete the request.

Every person in the human chain can be classed as a piece of middleware all the way from the house to the well and back again. If at any point someone enters the chain that doesn’t agree that the bucket is a bucket or doesn’t know how to handle it, then the it is dropped on the ground and the request fails.

Flexbox cross browser

Fri, 16 Aug 2013 00:00:00 +0000

Despite having been around for a while and having been through a couple of revisions, its support across browsers can vary greatly. From “Candidate Recommendation” on Chrome/Opera, “legacy flexbox” on Firefox and no support at all on IE9 and earlier.

Making flexbox work consistently across browsers was a challenge for us on a recent project, but I have found a solution that seems to work quite well.

Below is an SCSS @mixin that will attempt to handle compatibility between CR and legacy cross browsers flexbox.

@mixin flex($content: flex-start, $items: stretch, $direction: row, $wrap: wrap) {
 $packLegacy: $content;
 @if $packLegacy == flex-start {
 $packLegacy: start;
 } @else if $packLegacy == flex-end {
 $packLegacy: end;
 }

 $alignLegacy: $items;
 @if $alignLegacy ==flex-start {
 $alignLegacy: start;
 } @else if $alignLegacy == flex-end {
 $alignLegacy: end;
 }

 $oritentLegacy: $direction;
 $directionLegacy: normal;
 @if $oritentLegacy == row {
 $oritentLegacy: horizontal;
 } @else if $oritentLegacy == column {
 $oritentLegacy: vertical;
 }

/** SAFARI **/
 display: -webkit-box;
 -webkit-box-orient: $oritentLegacy;
 -webkit-box-pack: $packLegacy;
 -webkit-box-align: $alignLegacy;
/** FIREFOX LEGACY **/
 display: -moz-box;
 -moz-box-orient: $oritentLegacy;
 -moz-box-direction: $directionLegacy;
 -moz-box-pack: $packLegacy;
 -moz-box-align: $alignLegacy;
/** LEGACY **/
 display: box;
 box-orient: $oritentLegacy;
 box-direction: $directionLegacy;
 box-pack: $packLegacy;
 box-align: $alignLegacy;
/** IE 10+ **/
 display: -ms-flexbox;
 -ms-flex-wrap: $wrap;
 -ms-flex-direction: $direction;
 -ms-justify-content: $content;
 -ms-align-items: $items;
/** CHROME **/
 display: -webkit-flex;
 -webkit-flex-wrap: $wrap;
 -webkit-flex-direction: $direction;
 -webkit-justify-content: $content;
 -webkit-align-items: $items;
/** NATIVE **/
 display: flex;
 flex-wrap: $wrap;
 flex-direction: $direction;
 justify-content: $content;
 align-items: $items;
} //@mixin flex

@mixin flexItem($width) {
 -webkit-box-flex: $width;
 -moz-box-flex: $width;
 box-flex: $width;
 -ms-flex: $width;
 -webkit-flex: $width;
 flex: $width;

 min-height: 0;
}

Firefox however only half supports flexbox (all revisions) and to get around this I would recommend using Modernizr as this will add the class “no-flexbox” to the tag. This provides us with a simple work around that allows non flexbox supporting browsers render correctly by using specifically crafted and targeted CSS for non-flexbox browsers

I found that IE9 support could be implemented using the flexie javascript plugin. In IE8 Modernizr will add the class “no-flexboxlegacy” which can again allow you to create targeted CSS that wont affect your Flexbox layout.

For a great overview of the “CR” of flexbox, CSS Tricks has an amazingly comprehensive coverage of the functionality here http://css-tricks.com/snippets/css/a-guide-to-flexbox/

Creating Custom Routes in Silverstripe 3.1

Wed, 31 Jul 2013 00:00:00 +0000

We wanted to create a Route to our custom Products Controller in our products module for SilverStripe 3.1, such as: “http://www.examplesite.com/products/”

However looking at the Controller Documentation it was not clear how to create a route without an Action being supplied. In our example above the action is not specified, as we just want to use ‘view’.

Solution:

Create a /_config/routes.yml file containing the following:

---
Name: productsroutes
After: 'framework/routes#coreroutes'
---
Director:
 rules:
 'product': 'Product_Controller'
---

The above will redirect any Url that starts with “/product” to our Product_Controller. Note that everything after the rule, so after “/product”, is used in the next bit for matching.

Now we need to add private static $url_handers to Product_Controller to match our path, so in this example we need to match “$Slug!” which will match “”. Note the ! means the slug is required. Of course we want to direct this to a specific action, in this case “view”, this gives us:

private static $url_handlers = array(
 '$Slug!' => 'view',
);

Now just add “view” to the $allow_actions and add the “view” function. This gives the final Product_Controller as follows:

class Product_Controller extends Page_Controller
{
 private static $url_handlers = array(
 '$Slug!' => 'view',
 );

 private static $allowed_actions = array('view');

 public function view(SS_HTTPRequest $request)
 {
 // Your action code goes here
 return $this->render();
 }
}

Handy note:

You can put ?debug_request=1 on the end of your URL to see how it determines which Controller to use.

Disabling Cache in Silverstripe 3.1

Wed, 31 Jul 2013 00:00:00 +0000

While working with Silverstripe we found ourselves having to run “?flush=1” a lot to clear the Cache. To switch it off, while you work, add the following to your mysite/_config.php:

SS_Cache::set_cache_lifetime('default', -1, 100);

Set up SilverStripe 3.1 using only Git (No Composer)

Mon, 29 Jul 2013 00:00:00 +0000

We recently tried to use composer to set up SilverStripe 3.1, but ended up with a dependency nightmare. In order to work around this we decided to make use of Git submodules.

First set up your Git repository and run:

git init

Next set up a site directory for the code inside your Git repository. Then navigate to SilverStripe Installer in your browser and Download a copy. Extract files, and copy contents to site folder. Now we need to add the CMS and Framework. Navigate in a browser to the Git Hub repositories for CMS and Framework. Now copy the HTTPS clone URL for each project and run the following, to add these as Git sub modules.

git submodule add https://github.com/silverstripe/silverstripe-framework.git site/framework
git submodule add https://github.com/silverstripe/silverstripe-cms.git <path-to-site>site/cms

Now delete mysite/_config.php and load the site. Follow the normal install instructions displayed and you will have a running version of SilverStripe 3.1

Enabling MYSQL_CLIENT_INTERACTIVE with Doctrine 2 on Rackspace Cloud Database

Fri, 26 Jul 2013 00:00:00 +0000

We recently ran into problem using Doctrine 2 connecting to a Rackspace Cloud Database using the MySqli Driver.

Problem:

We have a long running PHP script that can sometimes run for hours at a time whilst processing information. This script requires a connection to a database, but has long periods of inactivity where there is no actual interaction with MySQL. By default MySQL uses the “wait_timeout” setting which states, how long an inactive connection can exist before it is killed. This is normally fine with web pages requests, as it is usually a short lived request. Unfortunately you do not have the ability to alter this setting when using Rackspaces Cloud Database.

Solution:

When using the MySQLi extension you can create a connection in “interactive mode” by passing the “MYSQLI_CLIENT_INTERACTIVE” flag, which will then use the “interactive_timeout” setting. On Rackspace this is set to 8 hours!

Annoyingly Doctrine does not allow you to pass any flags to the MySQLi Connection. So we overrode Doctrine\DBAL\Driver\Connection with our own Driver which then allows us to pass a “flags” parameter through.

Feel free to look at some of the other helpful features in we have added to Doctrine 2 here: ZucchiDoctrine

Installing PECL extensions for Zend Server 6

Mon, 13 May 2013 00:00:00 +0000

Recently we have revisited using Zend Server for some of our projects and decided to give the new version 6 a chance to prove itself.

Overall its a big improvement over version 5. There are still some things that are extremely annoying but we have decided that we can overlook them.

However there is one thing that we couldn’t do without. By default you will find that a number of PECL extensions will not install out of the box (at least this is what we experience using the Debian based install).

To fix this you will need to make sure you install the additional packages in ubuntu

php-5.4-source-zend-server or php-5.3-source-zend-server depending on the php version you are using
autoconf
build-essential

Once this is done you should now be able to install extensions from PECL without too much hassle.

Better Output for MySQL Select Command Using \G

Wed, 24 Apr 2013 00:00:00 +0000

If you ever find yourself using MySQL via command line and end up with something like this:

And thought there must be another way, well here it is: Use \G instead of ; at the end of your select command.

For example:

select * from CHARACTER_SETS\G

Below is an image of the output from this select:

Happy Querying!

Glorious Gluster - How to setup GlusterFS on Rackspace Cloud and Ubuntu 12.10

Fri, 15 Mar 2013 00:00:00 +0000

A few of our projects recently called for a distributed file-system that provided high availability and redundancy. After a tip off from a fellow techie and a quick browse around the net it appeared that a solution called GlusterFS appeared to tick all the boxes for what we were wanting.

However setting it up turned out not to be as trivial as I had originally anticipated. I’m going to try and put down the process we have evolved for setting it up on Ubuntu in the cloud

A couple of things to clear up first.

We are using Rackspace for our cloud but beyond the setup of the servers it should still be relevant
There are a number of ways to interact with Rackspaces set up but for this we are going to use the cloud control panel
We use Ubuntu as our preferred server which means that our config tends to be all over the place compared to other guides
You will need to set up a minimum of 2 servers and a separate block storage device for each.
We have set up and broken a few different variations of gluster setup so far and make no guarantees that the setup in this blog is infallable but its the best wehave so far.

Setting up the hardware

First things first. We are going to need to set up are some servers.

Feel free to create any size server you want. Just make sure to select Ubuntu 12.10 (or whatever version you may have that is newer).

You will also need to define a new network to work with. We use this to isolate the traffic between the nodes of our new gluster.

You can create a new network when creating the first of your servers. On the creation page under the networks heading you can find a “Create Network” button.

Hopefully this should be quite self explanatory. Now when you create subsequent servers you will then have the option to attach your new network (“GlusterNet” in my example).

Once the two starting nodes have been created then you need to add some additional block storage to store your data on. Make sure that you create blocks that have sufficient capacity for your needs. Something else to consider is using High Performance SSD storage. Its a little on the pricy side but well worth the expense if you are trying to eak out every ounce of performance from the implementation.

You will then need to attach one to each of your servers.

Once attached you will be able to see the details of the block mount point from the block storage details page.

Make a note of the mount point (in this case “/dev/xvdb”) as we will need that in a minute.

Prepare the Server

Now that we have a the hardware ready we can shell into a server to set it up.

First you need to shell into your server and update its OS as the images provided by most cloud supplier tends not to have the latest patches and updates. In our case it’s as simple as:

apt-get update 
apt-get upgrade

Once that’s done we then need to prepare the Block Storage device ( henceforth refered to as a “brick”)

if you run fdisk -l you should see that an entry that looks something like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdb doesn't contain a valid partition table

This indicates that our brick needs a partition table and formatting. We can achieve this be doing the following

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xe7da4288.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Partition type:
 p primary (0 primary, 0 extended, 4 free)
 e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): 
Using default value 209715199

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

I’ve highlighted the prompts and my responses. All we are doing here is creating a default partition table that has a single partition which uses up the whole disk.

now running fdisk -l should give us something that looks like

Disk /dev/xvdb: 107.4 GB, 107374182400 bytes
43 heads, 44 sectors/track, 110843 cylinders, total 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe7da4288

 Device Boot Start End Blocks Id System
/dev/xvdb1 2048 209715199 104856576 83 Linu

As you can now see we have a valid device of /dev/xvdb1 that we can mount_._ However we need to create a valid filesystem on the new brick before we can mount it. I have been doing this with Ext4 rather than XFS (which is the recommened filesystem from gluster), this is mainly down to the fact that when i tried using XFS I kept getting some issues with performance and access. I’m sure that with further investigation I could resolve this but as of yet haven’t had chance to. So far though I have had zero issues using Ext4. To create the filesystem we run:

mkfs.ext4 -j /dev/xvdb1

Next, create a folder to mount to, easily done by executing:

mkdir -p /glusterfs/brick

Finally, the simplest way to mount the device is via your /etc/fstab by adding the line

/dev/xvdb1 /glusterfs/brick ext4 defaults 1 2

and running mount -a as root (this will also mean that it mounts on boot for you automatically as well.)

Next we need to install the latest gluster version. At the time of writing this was v3.3.1. You can find a version to suit your OS at http://www.gluster.org/download. If you are using Ubuntu you can do the following

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-server glusterfs-client

By this point you will now have a single working server to continue on your going to need to set up your second server ready to create your new volume.

Once you have your second (or third, fourth, etc) setup its a good idea to add a reference to each one of them to your /etc/hosts file. This is not really necessary and you can just use the IP addresses of each server but it saves you having to remember each IP and makes it easier to identify.

Remember that we are going to be working with the new network interface you created earlier (i.e “GlusterNet”). to get the IP of your GlusterNet interface a quick ifconfig will show you an interface with an IP that matched the CIDR from earlier. In my case I now have 2 IPs of 192.168.3.1 & 192.168.3.2.

So now I add the following lines to my /etc/hosts file

192.168.3.1 gluster1
192.168.3.2 gluster2

Creating our volume

Now that the servers are prepared we can now play with the the tool gluster.This tool is a life saver in getting everything configured quickly and you can easily get a list of what its capable of by running gluster help. Now Im not going to take you through every command and option and would recomend reading the gluster manual to learn more.

What this tool actually does is help generate and manipulate all the required config that is then stored at /var/lib/glusterd/.

Firstly we need to tell gluster is that we have a pool of servers that will communicate with each other. Gluster refers to these as peers. To do this you need to run gluster peer probe gluster2 on each server for each server that will be used, replacing “gluster2” with the name names you defined in your /etc/hosts file. This will then create the appropriate files at /var/lib/glusterd/peers/

Now that all our peers have been defined we can get to actually creating the new distributed volume. This however requires a little consideration as there are some decisions you need to make.

If we take a look at the help for creating a new volume we can see that we need to decide on what options to use

volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>

- what are we going to name our volume
[stripe ] [replica ] - are we going to crate a striped or replicated volume and how many “bricks” are we going to create this volume with
[transport {tcp|rdma|tcp,rdma>] - What transport protocol do you want the peers to communicate with
- which servers/bricks do you want to use.

for more information on how to create you volume and what all the options mean have a look at these links

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Replicated_Volumes

http://gluster.org/community/documentation/index.php/Gluster_3.2:_Configuring_Distributed_Striped_Volumes

for our purposes we are going to run

gluster volume create myvolume replica 2 transport tcp gluster1:/glusterfs/brick gluster2:/glusterfs/brick

This now creates a new volume that spans both of our servers. you can confirm that this is the case by running gluster volume info and you should get something that looks like

Volume Name: myvolume
Type: Replicate
Volume ID: d3dd24fd-9482-44c3-9503-24291fad8193
Status: Created
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/glusterfs/brick
Brick2: gluster2:/glusterfs/brick

running this on both servers should give you the same results.

What you will now find is that the gluster command has created a plethora of files at /var/lib/glusterd/vols/myvolume/. As you work with gluster more and more you will find yourself drawn to these files as they control all the different aspects of how the volume works and performs. Most importantly we will need some information from these files when we come to configure a client to mount the volume.

All that is left to do now is start the volume which can be easily done with a quick gluster volume start myvolume

At this point we have now completed setting up our volume but we need to add some security. I would strongly recommend setting up a firewall using ufw to control access to the server. The easiest way to do this is to allow all traffic on your “GlusterNet” network interface as only the servers you attach to that network will have access. you can find a guide to using ufw at https://help.ubuntu.com/12.10/serverguide/firewall.html.

Mounting a Client

Now that we have a working volume we need to add some clients. To do this you will need to create a new server as above that is attached to the “GlusterNet” network but without the block storage (unless you really want it that is).

Make sure to add your gluster dfinitions to your /etc/hosts file

Once you have your new client server ready we can install the gluster client

apt-get install software-properties-common
add-apt-repository ppa:semiosis/ubuntu-glusterfs-3.3
apt-get update
apt-get install glusterfs-client

I’ve seen a number of different guides that tell you to install glusterfs-server as well but I have as yet had no need to as it all works without it.

Now there are a lot of way that you can mount your new Gluster volume. I have tried a few and have had varying results. What I have found is that the best way is to create a volume file. To do this we create a new file at /etc/glusterfs.vol.

volume gluster1
 type protocol/client
 option transport-type tcp
 option remote-host gluster1
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume gluster2
 type protocol/client
 option transport-type tcp
 option remote-host gluster2
 option remote-subvolume /glusterfs/brick
 option username <username>
 option password <password>
end-volume

volume replicate
 type cluster/replicate
 subvolumes gluster1 gluster2
end-volume

volume writebehind
 type performance/write-behind
 option cache-size 1MB
 subvolumes replicate
end-volume

volume cache
 type performance/io-cache
 option cache-size 400MB
 subvolumes writebehind
end-volume

What you will notice is that there is a and required for this to work. You can find these details on one of your peer servers in the file /var/lib/glusterd/vols/myvolume/trusted-myvolume-fuse.vol.

This /etc/gluster.vol file is basically going to inform the gluster-client software about how to connect to the gluster volume and all the available nodes to connect to. This provides us with some level of fail-over so should one node become unavailable the gluster client will seamlessly switch to a different one. It also allows us to define additional “translators” such as the performance-io one that you can see here. I would strongly recommend reading through the available translators to see which may be useful to you.

Now one of the main issues you will find with Ubuntu is that it will fail on boot if you try to add this mount to your fstab. To get around this you can use Upstart. if you create the following file at /etc/init/glusterfs-mount.conf making sure to change to the interface for your GlusterNet network (i.e. eth0 or eth1 or eth2, you get the idea)

author "Matt Cockayne"
description "Mount GlusterFS after networking available"

start on net-device-up IFACE=<interface>
stop on stopping network
stop on starting shutdown

script
    mount -t glusterfs /etc/glusterfs.vol /glusterfs
end script

As you can see we are using a straight mount command. The magic is that this will not be executed until the start clause validates which in this case is not until the network interface for “GlusterNet” is up and running properly. You will also see that we are mounting the /etc/gluster.vol file to /gluster (remember to create this folder to mount to) rather than mounting a network path as you might when mounting an NFS share.

If you wanted you could also add more to your upstart script to handle clean un-mounting of gluster thus allowing you to then use the service gluster-mount (start|stop|restart) commands

A quick reboot of the client server should confirm that it boots successfully and you will now end up with your volume mounted at /gluster. You can now test this by creating a new file. I tend to create an empty file at /gluster/mounted just so I have a quick reference that the folder is mounted. Once that’s created if you now go and take a look at the /gluster/brick on your “peers” you should see that there is now a file called “mounted” sat there looking all smug that it worked.

Caveats

Some important things for you to be made aware of

Never write directly to a brick. Make sure to write to the volume only through a configured client
Beware of split-brain. http://community.gluster.org/q/what-is-split-brain-in-glusterfs-and-how-can-i-cause-it/ http://www.gluster.org/2012/06/healing-split-brain/
RTFM - Read The F***ing Manual. Gluster is big and complex and there is a lot for you to understand. You can download a copy of the manual from here

Docblock, Oh Docblock, wherefore art thou Docblock (hint: Zend Optimizer Plus lost them)

Fri, 01 Mar 2013 00:00:00 +0000

tl;dr> I make a terrible assumption about Zend Optimizer+ and am corrected by Dominic in the comments;

Terrible post title I know but its the best I could come up with.

I’ve just come up for air after spending the majority of the day debugging some issues on our current development sandbox.

Now our sandbox tends to be quite bleeding edge in some circumstances and as such we run a fair few bits of unstable code. On the sandbox in question we have been running PHP 5.4.11 and unfortunately we have struggled to get APC working with it just the way we need it to. The lack of APC tends to make this sandbox quite slow.

We recently saw that Zend have open-sourced their OptimizerPlus extension (https://github.com/zend-dev/ZendOptimizerPlus) and that it was compatible with 5.4…. Fantastic, or so we thought.

So I added the new OptimiserPlus to the sandbox and everything was going swimmingly. That was until we had to run one of the utility scripts that we use to rebuild some of our data structures. These scripts make use of different parts of both Zend Framework and Doctrine which tend to rely on some heavy DocBlock annotations.

Now having used both APC and Zend Server knowing that they done affect this kind of functionality I had expected that OptimizerPlus would be fine…. Wrongo. It took me a good few hours of head scratching trying to figure out what had happened.

It turns out that OptimizerPlus suffers from the same flaws that eAccellerator does and strips Docblocks when caching the bytecode. This results in Reflection returning false when you call methods such as `getDocComment()`.

All in all its not the end of the world I just disable OptimizerPlus and have to wait till I can get APC working. Not my ideal scenario but I can live with it.

Something that does concern me is that there is currently an RFC that has gone to vote (https://wiki.php.net/rfc/optimizerplus) about integrating OptimizerPlus into the PHP 5.5 distribution. While this is great I do worry how many other things may break and will they be picked up and fixed for the 5.5 release.

Update: Since writing this post the RFC has finished being voted upon and has been approved. You can expect to see Optimizer Plus appearing bundled with PHP soon.

Update (15th Mar 13): Thanks to Dominics’ comment I now know that you can tell Optimizer+ to retain your Docblocks by setting your config using

zend_optimizerplus.save_comments (default "1")
	If disabled, all PHPDoc comments are dropped from the code to reduce the
	size of the optimized code. Disabling "Doc Comments" may break some
	existing applications and frameworks (e.g. Doctrine, ZF2, PHPUnit)

zend_optimizerplus.load_comments (default "1")
	If disabled, PHPDoc comments are not loaded from SHM, so "Doc Comments"
	may be always stored (save_comments=1), but not loaded by applications
	that don't need them anyway.

That’ll teach me to write a blog post without investigating more first.

Our Redmine install died, We all cried!

Sat, 23 Feb 2013 00:00:00 +0000

We have been using redmine for quite a long time and a few months ago attempted to upgrade from 1.3 to 2.something. Unfortunately I (quite typically) borked the installation and since then its been hobbling along after my attempts to fix it left it crippled.

Yesterday it finally gave up the fight and my attempts to resurrect the installation were futile. After a quick funeral (the eulogy was very touching), and wake in a nearby emporium of alcoholic beverages to commiserate our loss, I set about trying to figure out what to do next.

Alternatives

Now while Redmine is a worthy tool and has always managed to do what I needed in the past, recently its just not cut the mustard. I’ve kept toying with the idea of creating our own project management system but as with all in-house projects that we dream up its just never going to happen.

A quick google around our options are to either go for a hosted solution (not possible as we have some very specific requirements regarding our SCM that mean we have to host our own repos for client work) or Redmine (or chilli project).

Yes we looked at a number of other management tools and of them all Redmine is still the closes to what we needed.

Installation

So I spin up a new server instance of ubuntu 12.10 on the cloud and get to work installing the latest version.

As root I then run through these steps (you should assume that ALL of these steps require you to be root and files should be owned by root)

# update/upgrade base installation of ubuntu packages
apt-get update && apt-get upgrade

# install the requisite scm tools that we use
apt-get install git-core subversion mercurial cvs

# set up ruby
apt-get install ruby rubygems libruby ruby-dev

# set up apache & mysql
apt-get install apache2 libapache2-mod-passenger mysql-server mysql-client libmysqlclient-dev

# install imagemagick and the magick wand
apt-get install imagemagick libmagickcore-dev libmagickwand5 libmagickwand-dev

# create our user and database in mysql 
# replace uniquePassword with your own password
mysql -u root -p -e "create user 'redmine'@'localhost' identified by 'uniquePassword'"
mysql -u root -p -e "create database redmine"
mysql -u root -p -e "grant all on redmine.* to 'redmine'@'localhost'"
mysql -u root -p -e "flush privileges"

# clone redmine code to target location
cd /usr/local/share
git clone git://github.com/redmine/redmine.git

# set apache as the owner of redmine
chown -R www-data:www-data redmine

# move into our new redmine folder
cd redmine

# set up your database configuration
cp config/database.yml.example config/database.yml
vim config/database.yml

production:
 adapter: mysql2
 database: redmine
 host: localhost
 username: redmine
 password: uniquePassword

# install bundler gem
gem install bundler

# use bundler to set up redmine installation and without specified dependencies
bundle install --without development test postgresql sqlite

# set up our secret token
rake generate_secret_token

# set up our database and load default configuration
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake redmine:load_default_data

# edit /etc/apache2/sites-available/default
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
 <Directory "/usr/lib/cgi-bin">
 AllowOverride None
 Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
 Order allow,deny
 Allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log

 # Possible values include: debug, info, notice, warn, error, crit,
 # alert, emerg.
 LogLevel warn

 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# restart apache 
service apache2 restart

That should be enough for you to have a working installation of redmine ready for you to use/customise

Additional Config

We typically have additional steps that we would configure for our own installation.

# add plugin assets folder
mkdir /usr/local/share/redmine/public/plugin_assets
chown www-data:www-data /usr/local/share/redmine/public/plugin_assets

# enable some additional apache modules
a2enmod rewrite

# disable mod ssl
a2dismod ssl

# install gnutls 
apt-get install libapache2-mod-gnutls

# install ssl certificate bundle and key (this assumes that you have already copied the key and bundle to ~/)
mv ~/my_certificate.bnd /etc/ssl/certs/my_certificate.bnd
chmod 0644 /etc/ssl/certs/my_certificate.bnd
mv ~/my_certificate.crt /etc/ssl/private/my_certificate.key
chmod 0600 /etc/ssl/private/my_certificate.key

# now configure your /etc/apache2/sites-available/default-tls
<IfModule mod_gnutls.c>
<VirtualHost _default_:443>
 ServerAdmin webmaster@localhost
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 DocumentRoot /usr/local/share/redmine/public
 <Directory />
 Options FollowSymLinks
 AllowOverride None
 </Directory>
 <Directory /usr/local/share/redmine/public>
 Options Indexes FollowSymLinks MultiViews
 AllowOverride All
 Order allow,deny
 allow from all
 </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log
 # Possible values include: debug, info, notice, warn, error, crit, alert, emerg.
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/ssl_access.log combined

 GnuTLSEnable On
 GnuTLSCertificateFile /etc/ssl/certs/my_certificate.bnd
 GnuTLSKeyFile /etc/ssl/private/my_certificate.key
 GnuTLSPriorities NORMAL:!DHE-RSA:!DHE-DSS:!AES-256-CBC:%COMPAT
</VirtualHost>
</IfModule>

# Add some Rails / Passenger specific config to /etc/apache2/sites-available/default-tls
RailsEnv production
PassengerDefaultUser www-data
PassengerSpawnMethod smart
PassengerPoolIdleTime 300
PassengerMaxRequests 5000
PassengerStatThrottleRate 5
PassengerHighPerformance On

# change your /etc/apache2/sites-available/default to redirect to ssl
<VirtualHost *:80>
 ServerAdmin sysadmin@zucchi.co.uk
 ServerName mysite.co.uk
 ServerAlias www.mysite.co.uk
 RewriteEngine On
 RewriteCond %{HTTPS} off
 RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
 Options FollowSymLinks
 AllowOverride None

 ErrorLog ${APACHE_LOG_DIR}/error.log
 LogLevel warn
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

# enable your new default-tls vhost and restart apache
a2ensite default-tls
service apache2 restart

# setup &amp; configure email
# when prompted select "internet site" and enter the domain you are hosting redmine from i.e. mysite.co.uk)
apt-get install postfix

# create config file and uncomment the production settings for sendmail
cp /usr/local/share/redmine/config/configuration.yml.example /usr/local/share/redmine/config/configuration.yml
vim /usr/local/share/redmine/config/configuration.yml

production:
 email_delivery:
 delivery_method: :sendmail

service apache2 restart

#install pixel cookers theme cos we like it
git clone git://github.com/pixel-cookers/RedmineThemePixelCookers.git /usr/local/share/redmine/public/themes/pixel-cookers

Debug PHP CLI on Remote Server with Xdebug and PHPStorm

Wed, 06 Feb 2013 00:00:00 +0000

This was a head scratcher when I ran into this yesterday and I thought I would share my solution to the following scenario:

I need to debug PHP Command Line script, located on Remote LAMP Virtual WebServer running in Virtual Box with a Shared Folder, using local PHPStorm 5.0.

The solution:

You first must set PHPStorm to use remote file paths. To set these go to the following:

PHPStorm -> Peferences -> PHP -> Servers

This gives the following display:

Replace the Name, Host and Absolute path on the server, to match your own settings. Note keep the Name and Host the same for ease.

Next add some breakpoints in PHPStorm and set it to listen for any debug connections using the listener icon:

Now login to your Remote Server via SSH etc.

You now need to change settings for Xdebug in either xdebug.ini or php.ini depending on how you installed it. You also need to know the IP of the local machine. This can permanently set in the Network Setting of your VM in Virtual Box, so you will never have to change it. In my example the local machine running PHPStorm is:

192.168.56.1

Now edit the ini file that contains your Xdebug settings and set the following:

xdebug.remote_host = 192.168.56.1
xdebug.remote_connect_back = 0
xdebug.remote_port = 9000
xdebug.remote_handler = dbgp
xdebug.remote_mode = req
xdebug.remote_enable = 1
xdebug.idekey = phpstorm1

Be aware you might have to change the remote_host and the idekey based on your own environment. To better understand what each option does, see Xdebug Settings

Finally, when running the script you must set the following variables:

PHP_IDE_CONFIG="serverName=dev.example.com"

PHP_IDE_CONFIG will tell PHPStorm how to map the Remote File Paths to what it sees Locally. Again replace the URL with the Name/Host you set in PHPStorm. Note: You can export this, if your system is only running one site; mine is not.

You can run this inline with your script:

PHP_IDE_CONFIG="serverName=dev.example.com" ./testscript.sh

This should send you to PHPStorm where you earlier placed breakpoints.

Happy Debugging!

Quick and easy setup of and connection to NRPE on Ubuntu

Wed, 06 Feb 2013 00:00:00 +0000

About NRPE

NRPE (Nagios Remote Plugin Executor) is a useful tool that allows you to execute scripts on remote servers and return the output for ingestion by some form of monitoring software.

Setup

We currently have our own instance of Icinga running to monitor our servers and have recently started to offer access to it for our clients.

The majority of our servers (and our clients servers if we set them up) use one variant or another of Ubuntu. This means we can very quickly get our servers connected to a Nagios/Icinga instance.

First things first we need to install the nrpe server and all the associated plugins

apt-get install nagios-nrpe-server \
nagios-plugins-basic \
nagios-plugins \
nagios-plugins-extra

Next we need to edit the main nrpe config file to be found @ /etc/nagios/nrpe.cfg. What your looking for is the lines

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames 
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

allowed_hosts=127.0.0.1

# COMMAND ARGUMENT PROCESSING
# This option determines whether or not the NRPE daemon will allow clients
# to specify arguments to commands that are executed. This option only works
# if the daemon was configured with the --enable-command-args configure script
# option. 
#
# *** ENABLING THIS OPTION IS A SECURITY RISK! *** 
# Read the SECURITY file for information on some of the security implications
# of enabling this variable.
#
# Values: 0=do not allow arguments, 1=allow command arguments

dont_blame_nrpe=0

You will want to change this to the IP of your Nagios/Icinga instance and set the dont_blame_nrpe value to 1. Feel free to take a look round the rest of the file. Its all quite interesting and generally will documented. Be careful what you change though in case something breaks.

You will also want to look for some lines that are refererd to as “COMMAND DEFINITIONS” and look something like this

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

You can go ahead and comment these out as we will be adding our own definitions shortly. The main reason for removing these is that we will be configuring some specific scripts for our own use later that allow you to configure your requirements and thereshold from within your Nagios/Icinga config.

Configuration of Monitoring Server

Once this is complete you can now configure a new “check command” for use with your nagios/icinga server.

define command {
 command_name check_nrpe
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

define command {
 command_name check_nrpe_command_args
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
}

Here you can see that we have set up 2 different check commands. The first is a simple command requiring only one argument of $ARG1$ which would be the name of the command we want to run on the remote server. The second command is almost identical except for the fact it takes a second argument which allows you to input a series of “arguments” to be passed to the command on your remote server. each argument should be separated by a space.

Now that you have these you can then configure your hosts and services to make use of it. I would recommend having a trawl through the Nagios/Icinga sites & documentation to find out how to create a config that suits you.

Configuration of Remote Server

Now that we have our monitoring server ready its time to add the command we want to run to the remote server.

To do this your /etc/nagios/nrpe.cfg shoudl hopefully have a line in it that looks like

include=/etc/nagios/nrpe_local.cfg

if it doesn’t have a line like that then add it and edit the `/etc/nagios/nrpe_local.cfg` file to look a little like this

command[check_apt]=/usr/lib/nagios/plugins/check_apt
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p /dev/sda1
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

These are a few simple commands that I tend to use most often. These translate to your “check_nrpe” commands like so

$ARG1$ = everything inside the square brackets [ ]
$ARG2$ = each of the $ARG?$ keys as a single string separated by a space

Once that’s done you should be able restart your nrpe server with `/etc/init.d/nagios-nrpe-server restart`

It really is that simple. Do bear in mind that because you can pass arbitrary arguments into nrpe this was you could leave yourself vulnerable to a bit of maliciousness so its a good idea to make sure your firewall restricts port 5666 (the default port) to IPs you trust.

Compiling Apache 2.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

I’ve decided that I need to up my game when it comes to webservers. However I’m not yet ready to switch to Nginx or one of the other webservers out in the wild as I need something up and running rapidly.

Granted the numbers are definitely against Apache in a lot of benchmarks but historically I’ve always had a good experience and the entry level makes it much more appropriate for me to stick with it.

However Apache 2.2 is rather long in the tooth, thankfully 2.4 has been out for a while now. The problem I have is that I tend to favour Ubuntu as a platform and there is no sign of a 2.4 version appearing on the horizon anytime soon as they are waiting for it to be implemented upsteam in Debian before including it in Ubuntu.

Now there are PPAs available out there but im not overly happy using them (especially on production environments) So the only option is to compile.

First thing is to install all the dependencies we are going to need. Thankfuly ubuntu has a nice and simple way of handling this.

apt-get build-dep apache2

We can then download the source code and start the compilation.

So from the root of our new copy of the source we need to run our configure.

./configure --prefix=/usr/local/apache2 \
 --enable-mods-shared=all \
 --enable-http \
 --enable-deflate \
 --enable-expires \
 --enable-slotmem-shm \
 --enable-headers \
 --enable-rewrite \
 --enable-proxy \
 --enable-proxy-balancer \
 --enable-proxy-http \
 --enable-proxy-fcgi \
 --enable-mime-magic \
 --enable-log-debug \
 --with-mpm=event

You will notice that I’m installing it using the event mpm. Hopefully I’ll be covering more about the event mpm in the future.

Next we need to run make

make && make install

Once that’s complete you should be able to run

/usr/local/apache2/bin/apachectl start

and get the “it works” message through your webrowser when accessing the server IP.

Dont forget to configure apache to suit your specific requirements.

Something that will come up is how to start apache on boot. Seeing as Ubuntu uses Upstart it makes sense to utilise it for controlling apache.

So in the file `/etc/ini/apache.conf` we need to put

# apache2 - http server
#
# Apache is a web server that responds to HTTP and HTTPS requests.
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog

author "Matt Cockayne <matt@zucchi.co.uk"
description "Apache 2.4 HTTP Server"

start on runlevel [2345]
stop on runlevel [!2345]

console output

pre-start script
 mkdir -p /var/run/apache2 || true
 install -d -o www-data /var/lock/apache2 || true
 # ssl_scache shouldn't be here if we're just starting up.
 # (this is bad if there are several apache2 instances running)
 rm -f /var/run/apache2/*ssl_scache* || true
end script

# Give up if restart occurs 10 times in 30 seconds.
respawn limit 10 30
respawn

script
 if test -f /usr/local/apache2/bin/envvars; then
 . /usr/local/apache2/bin/envvars
 fi
 ULIMIT_MAX_FILES="ulimit -S -n `ulimit -H -n`"
 if [ "x$ULIMIT_MAX_FILES" != "x" ] ; then
 $ULIMIT_MAX_FILES
 fi

 /usr/local/apache2/bin/httpd -k start -D FOREGROUND
end script

This is a rather simple upstart script and I will be looking to update it at some point… but it works

Once that’s done you should find that on reboot Apache will start and take advantage of all the management features of upstart including attempting to respawn Apache should it end unexpectedly. You should also be able to then use the following commands to control Apache.

# how to start start apache
start apache
# or 
initctl start apache

# how to stop apache
stop apache
# or 
initctl stop apache

# how to restart apache
restart apache 
# or 
initctl restart apache

# check the status of apache 
status apache
# or
initctl status apache

I generally tend to avoid using the apachectl script found at /usr/local/apache/bin/apachectl once upstart takes control.

Compiling PHP 5.4 on Ubuntu 12.04

Tue, 06 Nov 2012 00:00:00 +0000

So recently I’ve been working with PHP 5.4 a LOT. Unfortunately Ubuntu (my main dev environment) is behind the times. So I’m resorting to compiling PHP manually.

Not a daunting as it may first appear. The really tricky part is working out your dependencies and `configure` script.

Hence the reason for this post as a reminder for myself and others that may want to do a quick compile. (I would recommend that if your compiling for a production/live environment that you make sure you understand what it is your compiling though before just using what’s here)

So where to start. Dependencies first I think

Ubuntu allows you to install dependencies for building source `apt-get build-deps`. We will use this and install any extras we may need.

apt-get install \
libxml2 \
libxml2-dev \
libssl-dev \
pkg-config \
curl \
libcurl4-nss-dev \
enchant \
libenchant-dev \
libjpeg8 \
libjpeg8-dev \
libpng12-0 \
libpng12-dev \
libvpx1 \
libvpx-dev \
libfreetype6 \
libfreetype6-dev \
libt1-5 \
libt1-dev \
libgmp10 \
libgmp-dev \
libicu48 \
libicu-dev \
mcrypt \
libmcrypt4 \
libmcrypt-dev \
libpspell-dev \
libedit2 \
libedit-dev \
libsnmp15 \
libsnmp-dev \
libxslt1.1 \
libxslt1-dev

And now the configure

./configure \
--prefix=/usr/local/php \
--with-apxs2=/usr/local/apache2/bin/apxs \
--enable-fpm \
--with-fpm-user=www-data \
--with-fpm-group=www-data \
--with-config-file-path=/usr/local/php/conf \
--with-config-file-scan-dir=/usr/local/php/conf.d \
--enable-debug \
--with-openssl \
--with-kerberos \
--with-zlib \
--enable-calendar \
--with-curl \
--with-curlwrappers \
--with-enchant \
--enable-exif \
--enable-ftp \
--with-gd \
--with-jpeg-dir=/usr \
--with-png-dir=/usr \
--with-vpx-dir=/usr \
--with-freetype-dir=/usr \
--with-t1lib \
--enable-exif \
--enable-gd-native-ttf \
--enable-gd-jis-conv \
--with-gettext \
--with-gmp \
--with-mhash \
--enable-intl \
--enable-mbstring \
--with-mcrypt \
--with-mysql \
--with-mysqli \
--enable-pcntl \
--with-pdo-mysql \
--with-pdo-pgsql \
--with-pgsql \
--with-pspell \
--with-libedit \
--with-readline \
--enable-shmop \
--with-snmp \
--enable-soap \
--enable-sockets \
--enable-sysvmsg \
--enable-sysvshm \
--with-xsl \
--enable-zip \
--with-pear \
--enable-zend-signals \
--enable-maintainer-zts

Once these are done then we follow the standard make process. Notice we are also running make test… very important as it givges more data for the developers to work with.

make && make test && make install

The next thing is configuring your php.ini file as the install doesn’t have one yet so we copy either the production or development default from the source code to the new conf dir and edit to suit your needs.

cp {php-source-dir}/php.ini-(development|production) /usr/local/php/conf

Thats it. All ready to roll… almost, this installation is the one I use for use with a webserver so you will want to add the appropriate directives to apache.

LoadModule php5_module modules/libphp5.so

AddHandler php5-script .php
AddType text/html .php

Rsync and custom SSH commands

Tue, 31 Jul 2012 00:00:00 +0000

Rsync is a great tool but can be a pain if you have to jump through hoops to connect via ssh such as connecting via a different port.

A simple solution is to use the -e flag (also knows as –rsh=COMMAND). This flag allows you manually define the ssh command to use when connecting

rsync -e 'ssh -p2020' -rav ./* user@server:

Will allow me to connect to a server with SSH listening on port 2020

Registering custom view helpers in ZF2

Wed, 18 Jul 2012 00:00:00 +0000

If you want to register custom view helpers with a module you can do so by using the service location built into the Skeleton Application and creating a module config that looks something like.

return array(
 'view_helpers' => array(
 'invokables' => array(
 // generic view helpers
 'truncate' => 'Zucchi\View\Helper\Truncate',

 // form based view helpers
 'bootstrapForm' => 'Zucchi\Form\View\Helper\BootstrapForm',
 'bootstrapRow' => 'Zucchi\Form\View\Helper\BootstrapRow',
 'bootstrapCollection' => 'Zucchi\Form\View\Helper\BootstrapCollection',
 ),
 ),
);

Bootstrapping ZF2 Forms

Tue, 17 Jul 2012 00:00:00 +0000

So…

With the release of beta 5 for Zend Framework 2 I thought it time for me to tidy up and fix a few modules I created back at beta 3.

Now I’m a big fan of Twitter Bootstrap CSS framework as I’m sure a lot of other people are as well. Seeing that the Zend Skeleton Application comes with bootstrap already included it was easy enough to set up my forms using the old ZF Forms found in ZF1.

However a brand spanking new Forms component has been rolled out with ZF2. The long and the short of this new component meant that I had the opportunity to hand roll a new way of making my forms work with Twitter Bootstrap.

So, a little tinkering, a quick pull request to ZF2 to allow the definition of arbitrary options and I came up with some useful View Helpers that can be dropped into a project and used.

You can find them at https://github.com/zucchi/Zucchi.

So how to use them. Lets start by creating a new form (we’ll keep it simple for now)

class MyForm extends Form
{
 public function __construct()
 {
 parent::__construct('myform');
 $this->add(array(
 'name' => 'price',
 'attributes' => array(
 'type' => 'text',
 'required' => 'required',
 'placeholder' => '0.99',
 ),
 'options' => array(
 'label'
 'bootstrap' => array(
 'help' => array(
 'style' => 'block'
 'content' => 'The price you wish to use'
 ),
 'prepend' => array('$'),
 'append' => array('¢'),
 ),
 ),
 );

 $actions = new Collection('actions');
 $actions->setAttribute('class', 'form-actions');

 $actions->add(array(
 'name' => 'submit',
 'attributes' => array(
 'type' => 'submit',
 'value' => 'Save',
 'class' => 'btn btn-primary'
 ),
 'options' => array(
 'bootstrap' => array(
 'style' => 'inline',
 ),
 ),
 ));

 $actions->add(array(
 'name' => 'reset',
 'attributes' => array(
 'type' => 'reset',
 'value' => 'reset',
 'class' => 'btn'
 ),
 'options' => array(
 'bootstrap' => array(
 'style' => 'inline',
 ),
 ),
 ));
 $this->add($actions);
 }
}

You’ll notice that I have highlighted some lines. Thanks to the ability to set arbitrary options we can define a “bootstrap” option which we can then use to allow us to pass data into our new bootstrap view helpers. You can also see that I have added a save and reset button to a collection. I’ll explain that later.

So what next… Rather than go into the mechanics of how to work with forms I’ll refer you to the ZF documentation and this excellent blog post

We then pick up by looking at your view, and the helpers I have created.

BootstrapForm($form, $formStyle)

One of the few things I miss from the ZF1 implementation of Forms is the self rendering aspect! So what did I decide to do? That’s right I created a view helper to render everything in one command.

The $this->bootstrapForm() takes two parameters. The first is quite obviously the form. The second is the style of form. This is directly related to the form types that can be found http://twitter.github.com/bootstrap/base-css.html#forms. You can use any of ‘vertical’, ‘inline’, ‘search’ & ‘horizontal’. If you dont specify a formStyle then it will default to ‘vertical’

Caveat: This helper will then iterate through all of the associated elements and render them first. Only after the direct elements have been generated will it then move onto Collections or Fieldsets (as soon as I work out how I’ll fix this).

BootstrapRow($element, $formStyle)

This is a straightforward modification of the FormRow helper that come bundled with the new component.

We have a few differences now though. We have a second parameter as with the BootstrapForm view helper and the output is generated using sprintf and a set of templates that mimic the structures of the different form styles from bootstrap.

This helper can be used by itself to generate an element row and is used by the BootstrapForm helper

We can also now take advantage of the “bootstrap” options we set earlier.

Bootstrap Options

style

The style of form element to use regardless of what style may be passed into the view helper (you can see an example of this in the buttons from the MyForm example above)

help

This works in the same way as “description” did from ZF1 but allows you to define it either as a string or an array with the keys “style” for either ‘inline’ or ‘block’ and “Content” which should be self explainatory

prepend

Takes advantage of Bootstraps ability to prepend blocks to an input field. This can be defined as a single string, or an array of strings to allow you to add multiple blocks should you want to

prepend

Takes advantage of Bootstraps ability to append blocks to an input field. This can be defined as a single string, or an array of strings to allow you to add multiple blocks should you want to

These options get evaluated and spat out from the new renderBootstrapOptions() method as part of the “render”.

BootstrapCollection($element, $style, $wrap)

Again this is a direct rip off of the FormCollection helper found in the ZF2 Form component witha few modifications. The main difference is that is makes use of the BootstrapRow helper and has methods and properties to allow the setting of the form style to use.

You can see from the MyForm example above that we set a Collection called ‘actions’. This is a pretty standard way of grouping elements together. You can also see that we set a class for the Collection which may look familiar to those that have used Twitter Bootstrap for a while.

What our helper will then do is wrap the buttons in a div with the appropriate class attached. If you were to define a label for the Collection/Fieldset You would then also find that the fieldset and legend tags are also spat out with our <div class="form-actions"> sandwiched between them and the elements.

Result

So what we now get when we use MyForm with out helpers.

$this->bootstrapForm($form, 'horizontal');

Should now look something like this

How you can use it

As of right now you can get the library from its repo on github @ https://github.com/zucchi/Zucchi and can be found on packagist for use with composer

Edit: The bootstrap stuff has moved to a new location as a separate ZF2 module. you can find it @ https://github.com/zucchi/ZucchiBootstrap or @ packagist for use with composer

Loaded Testing

Sat, 30 Jun 2012 00:00:00 +0000

I recently had to do some load testing for a site recently that would allow me to test in excess of 100k requests in a 60 second period…

So I decided to do some testing using JMeter as it seemed like a suitable tool for doing what I needed and I had used it for some simpler testing in the past.

After a little fumbling around I managed to get a test plan designed that would simulate 10k users actually navigating the site and adding to a cart etc, with a number of various interactions. It wasnt perfect but it would correctly simulate over 100k requests.

So feeling quite pleased with myself I started the test from my laptop. Now I’m not a big gamer, I’m known to play a little World or Warcraft from time to time but that’s about it. So when it comes to computing power i tend to opt for battery life over sheer grunt.

Suffice to say, my laptop fell flat on its face, and if it hadn’t it turns out that the connection I was using just wasn’t up to the task of handling that much traffic adequately.

So plan B…

I quickly fired up the largest AWS instance available and got a copy of jmeter installed. A little tinkering with my test plan and some googling on how to run jmeter without a gui and a quick

./jmeter -n -t test-plan.jmx

and it appeared to be running.

(Please bear in mind that I’m being overly kind… it took a LOT of tinkering and twice as much Googling to work out how to get the test results out so i could actually get some idea of WTF was happening during the test)

So… client “happy”… I decided to go and find a better way to do my load testing in the future.

Sticking with JMeter I managed to find this gem of a page

http://jmeter.apache.org/usermanual/remote-test.html

tl;dr > use your local install of jmeter to trigger tests to run on one or more remote “nodes” and then have all the results sent to your local install.

So I set to work!

Building a Node

First I need to set up an AWS instance that we can use and duplicate so I can quickly build a cluster of nodes on demand. I’m a big fan of Ubuntu so I spin up a micro instance of 12.04 server. Next I shell into the instance and install the default Java runtime from apt

apt-get install openjdk-7-jre

Yes I know there are other more appropriate runtimes, but i dont really care… i just need it to work and it does.

next I grab a copy of the latest stable from http://jmeter.apache.org/download_jmeter.cgi and un-tar it to /usr/local/jmeter

(N.B. JMeter is available through the apt but I had issues with that version and you need to make sure that both your local version and all the nodes run the same version of jmeter)

We can now test that the install is working running /usr/local/jmeter/bin/jmeter-server and you should get some output that looks similar to

Created remote object: UnicastServerRef [liveRef: [endpoint:[10.???.???.???:38939](local),objID:[46522b57:138381f1023:-7fff, 2635011707874933136]]]

Which tells us that the server is running.

BUT unfortunately its not going to work just yet. Because we are using Amazons EC2 we are going to relying on their NAT for routing. Out of the box JMeter just wont work properly.

However there is something we can do to combat this. We can set the parameter RMI_HOST_DEF that the /usr/local/jmeter/bin/jmeter-server script will include in starting the server.

export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

I’ll explain what we are doing here. Amazon have been quite clever by providing a meta-data endpoint that you can poll from within your instance to get key pieces of data… Including the public dns record.

We can use this endpoint and using wget pipe that into the RMI_HOST_DEF param (ensuring that we prepend -D) and then export that so it becomes available to the /usr/local/jmeter/bin/jmeter-server script.

Now to get the server to start on boot.

a quick upstart script should solve this

# Upstart script to initialise jmeter-server

description "JMeter Server"
author "Dev in Charge "

start on started networking
stop on stopping networking
stop on stopping shutdown

console output

script
 # get the current public DNS record
 export RMI_HOST_DEF=-Djava.rmi.server.hostname=$(wget http://169.254.169.254/latest/meta-data/public-hostname -q -O -)

 # start jmeter in server mde
 /usr/local/jmeter/bin/jmeter-server
end script

saving this to /etc/init/jmeter-server.conf will mean that it will auto-start jmeter-server on boot and allow you to manually control the process using start jmeter-server and stop jmeter-server

and thats it… instance configured

All you need to do now is save the instance as an AMI and you have an on-demand image for spinning up a cluster of remote JMeter servers for you to play with.

Configuring your local installation

Now that the server side is working we need to configure our local installation to allow it to connect.

First things first however, make sure you are using the same version of JMeter as you are running on the server.

We need to edit the jmeter.properties file that can be found in the bin folder of the installtion you downloaded. Look for the parameter remote_hosts This needs to be set with the public dns of the remote server(s) your connecting to. for example

remote_hosts=ec2-176-34-164-170.eu-west-1.compute.amazonaws.com,ec2-123-34-456-789.eu-west-1.compute.amazonaws.com

Thats your local version configured. You will now be able to tell your local version to run tests on any or all of your specified remotes.

However if your like me you work behind a router/firewall. If so this isnt the end of the story. When you send a test plan to a remote from your local install it will also send the IP address of your local machine for it to send the results back to. JMeter does this by looking up where your current hostname resolves to. In my circumstance it resolved to 127.0.1.1. The reason it did this is down to the fact my systems host file had the line

127.0.1.1 devincharge.local

To resolve this I had to change it to my external IP address

89.345.871.79 devincharge.local

And set up port forwarding from my router to my local machine for all ports from 1024 to 65535. Now, you can if you want use specific ports so you dont have to port forward everything from your router, but i’ll leave that for you to lookup as there are plenty resources on how to do this for you to google and I’ve waffled on for far too long already.

Happy testing

Orienteering on PHP Boy Scout

Sign your own binaries with go-tool-base, part 7: rotation and break-glass

Why there’s no auto-rotate button

Rotate by minting a new key, never by changing the old one

The dual-publish, dual-sign window

The break-glass key, for when there’s no handover

The whole chain, end to end

Sign your own binaries with go-tool-base, part 6: sign every release with GoReleaser

One command wires the signing in

Why only the checksums

No shim, just gtb

Where the credentials come from

Don’t sign when there’s nothing to sign with

Cut a release

A two-person gate, if you want one

Where this leaves you

Sign your own binaries with go-tool-base, part 5: embed the key and require verification

Enable signing with one command

Drop your key in

The WKD cross-check, via --email

Confirm the cross-check is actually firing

Require it, in the right order

Where this leaves you

Sign your own binaries with go-tool-base, part 4: mint and publish your public key

Mint the public key from KMS

Mint the rotation authority while you’re here

Build the WKD tree

Publish it somewhere the platform can’t reach

Where this leaves you

Sign your own binaries with go-tool-base, part 3: keyless CI signing with OIDC

Where the identity provider comes from

The GitLab path

The GitHub path

The gotcha that wastes an afternoon: token audience

Where this leaves you

Sign your own binaries with go-tool-base, part 2: a signing key in AWS KMS

What you’ll need first

The module

What you just built

Four principals, one key policy

The outputs you’ll carry forward

What this costs

Where this leaves you

Sign your own binaries with go-tool-base, part 1: sign and verify on your laptop

Make a signing key

Mint the public key you’ll actually publish

Sign something

Verify it, two ways

Where this leaves you

Sign your own binaries with go-tool-base

What you’re protecting against

Two paths through it

Before you start

The parts

Generate a command from a script or a sentence with go-tool-base

Two ways in, the same files out

From a script: csv_stats.py becomes csv-stats

From a sentence: a URL health-checker

What “self-repair” actually means

Flags you declare, logic it writes

Where that leaves you

Building a web service with go-tool-base, part 6: seeing what your service is doing

The three things you actually want to see

Turn it on

Wire it onto the transports

Go deeper than the plumbing

Point it at a collector

One request, the whole way through

A word on the other telemetry

Where that leaves the service

Building a web service with go-tool-base, part 5: docs that write themselves

The annotations, a third time

Generate the OpenAPI document

Serve it, viewer and all

Why it goes on the same server

One proto, three faces

Building a web service with go-tool-base, part 4: REST for free, with the gateway

What the gateway actually is

Map HTTP onto the proto

Generate the gateway

No shim, just `gtb`

The WKD cross-check, via `--email`

From a script: `csv_stats.py` becomes `csv-stats`

A page from `html/template`

Why not just `net/http`?

What `update` does, step by step