The first time the job ran on a real instruction, the instruction failed and the job printed this:

failed    cicd/wikis/specs/0086-a-project-that-names-its-own-assets — 404 Not Found: not found

1 of 1 needed attention. The merge stands; nothing here failed the pipeline.
Job succeeded

It told you, very politely and in so many words, that the thing it was there to do hadn’t happened and that it wasn’t going to make a fuss about it.

A job to stop specs lying

Some context, because the job matters more than it sounds. Every project in the estate carries its specs on a wiki, and a spec has a status: draft, approved, implemented. When the CI/CD project went and counted, 59 of its 82 specs were sat at APPROVED with the work already shipped, which is to say most of the status fields in that wiki were wrong, and they were wrong in the direction that makes a reader think work is still to do.

So a job was built to fix it. When a merge request carries a trailer saying which spec it implements, colophon’s apply verb runs after the merge and moves that spec’s status. The report above is its first go at the real thing. A 404, and a green pipeline, and the status stayed exactly as wrong as it had been. The only reason anyone found out was that someone opened the trace to see whether it had worked.

The session that found it raised a ticket on colophon, and a spec to fix it was on the wiki within half an hour.

The careful fix

The spec was careful, and the caution was reasonable. It laid out three ways a caller could learn that an instruction had failed: a non-zero exit code, an opt-in flag, or a machine-readable summary. Then it recommended the flag. --fail-on-problems, off by default, exiting with code 2 when an instruction had failed.

The reasoning was a constraint it had written down first: whatever changed, a project that upgraded colophon and touched nothing else must not start failing. A release that silently turned default branches red across the whole estate is not one anybody would thank you for, and pre-1.0 or not, that’s a fair thing to worry about.

I read it, and six minutes after the page went up I was asking why we weren’t just exiting 1 on error. Why have an exit code of 2 at all? And while I was in there, was it going through the framework’s error handling to set the code, or exiting directly from the command? Because exiting directly bypasses the CLI middleware, and that would want fixing whatever else we did.

Then, five minutes later, the question that took the flag apart: why did colophon not already exit 1 on problems? Why had we decided, in the first place, to exit 0 when something had failed?

Where the zero came from

Nine days earlier, as it turned out, and I’d signed it off.

The apply verb’s own spec had a decision in it called “it never fails the pipeline”, and the reasoning is still sound on its face. By the time apply runs the merge has happened, and a red pipeline because an annotation didn’t post is worse signal than a yellow one. A red default branch that can’t be acted on is how red stops meaning anything. That decision cited the estate’s release-stamping job as its precedent, and the code carried the comment faithfully:

// Never a non-zero exit for a target that failed. By the time this runs
// the merge has happened, and a red pipeline over an annotation is worse
// signal than a yellow one (spec 0012 D5).
return out.err

So I went and read the precedent. The release-stamping script raises on anything below an HTTP 500, 404 and 403 included, and an uncaught exception exits 1. The Discord release job runs exit 1 when Discord rejects the post. Neither of them exits zero. What both of them do is set allow_failure: true in their CI component, so a failure shows yellow in the pipeline instead of red.

The precedent said the opposite of what had been taken from it. “Never fail the release” had been translated into “the binary exits zero”, and that is a different and much stronger claim.

Yellow was what the original decision wanted all along, and exit 1 with allow_failure on is how you get it. Exit 0 got green instead, which is neither what it asked for nor true. Two questions had been collapsed into one: should the pipeline go red and can the caller know. Answering the first with no had answered the second with no by accident, and a flag defaulting to off would have kept that accident as the default.

Yellow is acceptable. It means the thing completed and there may be items for an engineer to check. That was the whole of my answer, and the spec went to approved on it.

Not every problem is a signal

With no flag, the next question was what should turn the exit code non-zero, and that one wasn’t obvious.

apply counts two kinds of trouble. Failed means the forge refused: a 404 from a mistyped page, a 403 from a token without the scope. Actionable, and usually somebody’s mistake. Skipped means the forge can’t do the thing at all. Bitbucket has no wiki, Gitea’s adapter has no comment capability, and that’s a fact about the forge, not about the commit.

Gate on both and a repository on Gitea reports a problem on each merge, for ever, that no change to the repository could ever clear. That’s the definition of a signal people learn to ignore, and it would have made the exit code useless on three of the four forges colophon supports. So only Failed turns the exit non-zero, and the summary line still counts both, because a skipped instruction is worth a person’s attention once.

Through the front door

My other question, the one about the error handler, turned into a decision of its own. The verb returns an error from its command and does nothing else. There’s no os.Exit anywhere in colophon.

That matters more than it looks. The framework’s root Execute does four things after the command tree returns, in order: it runs cleanup, which stops the config-watcher goroutines; it asks the error handler for the exit code; it flushes telemetry; and then it exits. An os.Exit inside a command skips all four, including the signal-aware path that exits 128 plus the signal number. None of that shows up in a passing test, which is why it’s written into the spec instead of left for whoever’s next in that file with a deadline.

It’s a breaking change, so say so

A repository running apply without allow_failure had a green pipeline before this and a red one after it. That’s the correct behaviour and it’s still something a person can be ambushed by, so it shipped with a Release-Note: trailer on the commit naming allow_failure as the line that restores the old colour. Colophon is below 1.0, and a breaking-change footer down there cuts v1.0.0, which this didn’t deserve.

Building it turned up two small things. The spec had promised no output would change, and it had to, because the summary line said “nothing here failed the pipeline”, and after this change that sentence was false. It now says which of the two happened. And three tests were pinning the old decision, two of which had been written an hour earlier for an unrelated ticket and used require.NoError because that had been true when they were written.

The one that stays at zero

The counter-example is in the same tool, and it’s what stops this becoming a slogan. colophon’s announce verb, which posts a release to chat channels and to this blog’s changelog, still exits zero when an adapter fails. On purpose. By the time it runs the release is already public, nothing downstream gates on whether Discord took the message, and no caller is waiting to find out.

apply gets the non-zero exit because a job wants to gate on it. announce doesn’t, because nothing does. That’s a less satisfying rule than “always exit non-zero on failure”, and I think it’s the truer one. Report and carry on, by all means… just don’t report and succeed.