Featured image of post Bringing Cloudflare in from the cold

Bringing Cloudflare in from the cold

Everything on AWS here runs on credentials that last minutes. Cloudflare I'd been clicking by hand, and bringing it in meant living with a long-lived secret and making it as small as I could.

Whatever this estate runs on AWS is described in OpenTofu, and there’s no long-lived AWS key anywhere in it. A pipeline assumes a role by OIDC, gets credentials that last minutes, and does its work. I’d not go back.

Cloudflare was not in any of that. Cloudflare I clicked, in the dashboard, like it was 2014.

Why it was outside on purpose

It wasn’t an oversight.

Cloudflare in this estate does DNS for sixteen domains, email routing for all of them, and it serves the public signing keys that people use to verify the releases I publish. That last one is why it sat outside the automation. If the build system can’t reach the place the public keys live, then a compromised build system can’t quietly replace a public key, and the trust chain that verification depends on doesn’t run through CI at all. Keeping it manual was the control.

Then the release binaries needed somewhere permanent to live, R2 turned out to be the sensible place to put them, and R2 is Cloudflare. So the question arrived whether I liked it or not: bring Cloudflare under infrastructure-as-code, or keep clicking.

What I told the session working on the release store was that we could separate the keys out with permissions easily enough, and that hopefully there’d be an OIDC equivalent to what AWS does.

There isn’t. Cloudflare has no OIDC federation for its API. So onboarding it meant accepting that a long-lived secret would exist, and the whole of the rest of that day was about making that secret as small as it could possibly be.

One token with godlike powers

I went and looked at what the existing token could do first, and couldn’t remember, which is its own answer. If it was the same token I use for local development then it had far too many permissions for a pipeline, so the sensible move was to stop reasoning about it and recreate the lot properly.

Halfway through, I realised the tokens I was looking at were user tokens, attached to me, and there’s a separate account-level section I’d missed entirely. Account tokens are the right shape for this. They belong to the account rather than to a person, so a token doing a machine’s job isn’t inheriting everything my own login can do.

What came out of it was a token per job. Infrastructure gets DNS and email routing across the account, because that’s what it manages. The release pipeline gets one that can write objects to the release bucket and nothing else, and I tightened that one again a couple of hours later to make sure it was the one bucket. Another project that deploys its own site gets what that deploy needs on its own zone. This blog gets zone and analytics reads, which is all its weekly report wants. And the token on my laptop keeps its broad permissions, because that one is attached to a person at a keyboard who can be asked what they’re doing.

Then a second thought, before anything ran: the infrastructure token had been going in as a group-level variable, available to every project in the group, and that felt risky with those permissions on it. It moved to a project-level variable on the one project that manages infrastructure. Old tokens deleted, new development token generated, and the whole thing re-pointed.

None of that is clever. It’s the difference between one credential that can do the lot and several that can each do one thing, and it took a morning.

A pipeline with no AWS identity

One real change came out of this on the public side. The cicd component that runs OpenTofu was built for AWS and assumed an OIDC role every time, and a Cloudflare-only stack has no AWS identity to assume. It now runs a stack whose providers aren’t AWS at all: leave the role ARN empty, no web identity is exported, and the providers authenticate from their own environment variables.

That’s a small input and a useful one, because “our IaC pipeline only works for one cloud” is a constraint you don’t notice until the second one turns up.

Adopting a live account without touching it

Then there was importing sixteen domains’ worth of DNS and email routing that already existed and were serving live traffic.

The rule I gave myself was that the plan against the live account had to come out as a no-op. Not “a few harmless changes”, not “it’ll just reorder some records”. Zero. Anything else means the code doesn’t describe what is really there, and the first apply would find out which of us was wrong in front of everybody’s email.

So it went zone by zone: import the real state locally, run the plan, fix whatever the provider represented differently from how I’d written it, move on. Batches per domain to keep the blast radius small, applied locally at each turn, then the whole lot stacked into one merge request to review as a piece. The provider had its own sharp edges in there, including a resource that can’t be imported at all and has to be created through the code.

By the end the plan was clean, the batches were stacked into one merge request, and the account was written down instead of remembered.

What happened to the airgap

Which brings me back to the keys.

The part of the key’s publication that I kept manual is still manual. The site that serves it is deployed by hand, from my machine, with a token that can do that and nothing else, and no token in CI has any Pages or Workers permission at all. The build system can’t publish a key.

What I didn’t think through that day is that the infrastructure token can edit DNS on every zone, including the one the key’s hostnames live in. A name that points at the key site can be pointed somewhere else, and for someone asking that name for my key, that’s as good as changing what it serves. So the airgap is narrower than “the build system can’t touch the key”. It’s “the build system can’t publish a key, and it can redirect the name people ask for one”.

That’s a real gap, and it’s there because I scoped the token by what the stack needed to manage, and the stack manages those records.

Narrowing it again

It didn’t stay that way for long. The token that every plan sees can now only read, and the one that can edit DNS reaches a single deploy job that only runs from a protected tag. So pointing the key’s hostnames somewhere else needs code on a protected branch of the infrastructure project and a deploy run from a release tag, or my own token from my own machine.

A rule about who can change something doesn’t tell you whether it changed, though, so there’s also a check that runs every six hours with no credentials at all. It asks those hostnames what keys they’re serving and compares the answer against digests written down in the repository. Repoint a name and serve everyone a different key, and that check goes off.

It’s worth being clear about how far that goes. It doesn’t make the key independent of CI, it doesn’t mean a person signs off every deploy, and a check that sees what my own runner sees won’t spot an attack aimed at one particular person. What it does is put the name people ask for my key behind the same door as the code that manages it, and keep an eye on what that name hands out… which is a lot closer to what I thought I had that day.

Built with Hugo · Theme Stack designed by Jimmy