AWS on PHP Boy Scout

Wire up an AWS billing alarm before the surprise

Tue, 14 Jul 2026 00:00:00 +0000

Everybody who has run anything on AWS carries the same low-grade dread: the bill you don’t see coming. A cron that spins up an instance and forgets to kill it, a misconfigured NAT gateway billing you by the gigabyte, a test that leaves a fleet running over a bank holiday weekend. The fix is well known and takes ten minutes: set a budget, and have it shout at you when spend crosses a line.

The trap is subtler, and it’s the reason for this post. You can set that budget, wire it to a topic, see the notification sitting in the console exactly as you configured it, feel thoroughly responsible… and then hear nothing at all on the day spend sails past your line. A billing alarm that doesn’t fire is worse than no alarm, because you’ve stopped watching for the thing yourself. So here’s how to wire one up properly, and the one piece AWS will actively nudge you into getting wrong.

The easy part: the budget

A budget is one resource. Give it a monthly cost limit and a threshold to shout at:

resource "aws_budgets_budget" "monthly_cost" {
 name = "monthly-cost"
 budget_type = "COST"
 limit_amount = "50"
 limit_unit = "USD"
 time_unit = "MONTHLY"

 notification {
 comparison_operator = "GREATER_THAN"
 threshold = 80
 threshold_type = "PERCENTAGE"
 notification_type = "ACTUAL"
 subscriber_sns_topic_arns = [aws_sns_topic.alerts.arn]
 }
}

You can point a notification straight at an email address, and plenty of people do. I’d send it to an SNS topic instead, because a topic is a fan-out point: today it emails you, tomorrow it also pokes Slack, a Lambda, or your on-call rota, and you change none of the budget wiring to do it. If you’ve built anything security-conscious you probably already have such a topic. The public terraform-aws-security-baseline module, for instance, stands up an alerts topic wired to shout about GuardDuty findings and root-account activity. Hanging your cost alarm off the same wire, so everything that matters arrives in one place, is a tidy instinct.

It’s also exactly where the trap is waiting.

The hard part: encryption locks the alarm out

A well-built alerts topic is encrypted, and it should be. The security-baseline one uses a customer-managed KMS key, so nothing sensitive ever sits in plaintext on the topic. Encrypt the topic, point your budget at it, and nothing about the setup will look wrong. The budget is real, the notification is configured against the topic, terraform apply is happy. But the alarm won’t fire, and AWS is going to be spectacularly unhelpful about why.

Here’s the mechanism, because it’s worth understanding rather than just pasting the fix. Most things that feed an alerts topic arrive through EventBridge, which publishes on your behalf. A budget doesn’t. AWS Budgets publishes to SNS directly, as its own service, under the principal budgets.amazonaws.com. To put a message onto an encrypted topic, whoever’s publishing has to encrypt it first, and that means being allowed to use the topic’s KMS key. EventBridge was granted that. The budget service was not.

You can see the grant the topic already makes in the module’s key policy: EventBridge is named as a user of the key, allowed to generate a data key and decrypt so it can encrypt what it publishes (alerts/main.tf). There’s no equivalent line for budgets.amazonaws.com, because the module was built for security findings, not cost. So the budget goes to publish, can’t touch the key, and the alert never leaves the building.

And here’s the bit that turns a five-minute fix into a lost afternoon: AWS’s own guidance points you the wrong way. Try to attach a budget to an encrypted topic in the console and it stops you with “The SNS topic is encrypted. The SNS topic won’t work without additional permissions. Disable encryption on the topic.” Read that twice. Faced with an encrypted topic a budget can’t use, AWS’s headline suggestion is to turn the encryption off. Don’t. You don’t rip out a good security control to make a billing alert work. And the belated warning you might get when a budget can’t publish, an out-of-band note asking you to make sure budgets is on the topic’s list of allowed publishers, points straight back at the topic’s access policy. Neither message ever mentions the key. Every signal AWS hands you is aimed at the wrong thing, and the real fix is to grant the alert access to the key, exactly the way the module already did for EventBridge.

The two grants that fix it

The budget needs the same treatment EventBridge got, in two places. First, on the KMS key policy, name the budget service as a user of the key (AWS documents this exact requirement):

statement {
 sid = "AllowBudgetsPublishViaTopic"
 effect = "Allow"
 principals {
 type = "Service"
 identifiers = ["budgets.amazonaws.com"]
 }
 actions = [
 "kms:GenerateDataKey*",
 "kms:Decrypt",
 ]
 resources = ["*"]
}

Second, on the SNS topic policy, let the budget publish:

statement {
 sid = "AllowBudgetsPublish"
 effect = "Allow"
 principals {
 type = "Service"
 identifiers = ["budgets.amazonaws.com"]
 }
 actions = ["sns:Publish"]
 resources = [aws_sns_topic.alerts.arn]

 condition {
 test = "StringEquals"
 variable = "aws:SourceAccount"
 values = [data.aws_caller_identity.current.account_id]
 }
 condition {
 test = "ArnLike"
 variable = "aws:SourceArn"
 values = ["arn:aws:budgets::${data.aws_caller_identity.current.account_id}:*"]
 }
}

Those two aws:SourceAccount and aws:SourceArn conditions are AWS’s documented shape for the budget topic policy, and they’re worth keeping: they stop any other account’s budget from ever using your topic (the confused-deputy guard the baseline already applies to its EventBridge statements). The pair of grants above are otherwise twins of what the module writes for EventBridge, with one principal swapped. Add them and the message flows.

One thing that catches people out: this only works with a customer-managed key. If your topic uses the default AWS-managed aws/sns key, you can’t edit its policy to add a service principal, so a budget can never publish to it. That’s a large part of why the security-baseline module brings its own CMK, and it’s the difference between “encrypted” and “encrypted and actually usable by the things that need it”.

Watch it fire once

The most important step is the one people skip, and it’s the whole reason I wrote this down. I won’t believe an alarm works until I’ve watched it fire, and a billing alarm is no exception. Temporarily drop the threshold to something you’re certain you’ve already crossed this month, a single percent if you like, apply, and wait for the message to actually land in your inbox or your Slack. When it does, you know the whole chain is live: budget, topic policy, key policy, subscription. Then put the threshold back.

It takes five minutes and it’s the difference between an alarm and a decoration. The budget was always the easy 80%. The KMS key policy is the fiddly 20% that decides whether, on the day a runaway resource starts eating your account, the thing pages you or just watches.

The security service I had to switch off

Mon, 01 Jun 2026 00:00:00 +0000

A while back I wrote about hardening the account that would hold the signing key, and one line in it has aged badly. “GuardDuty is already looking,” I wrote: the account watched from day one, threat detection on before the key even arrives. Then I went to apply that baseline to a brand-new account, and GuardDuty wasn’t looking at all, because the account wouldn’t let it.

What the baseline switches on

The baseline runs a threat-detection module that turns on three AWS services: GuardDuty, Security Hub, and IAM Access Analyzer. Each sits behind an enable_* toggle, all defaulting to on, which is the right default. An account about to hold something sensitive should be watched, flagged and analysed from the start. The hardening post’s whole argument was that you fit the locks before you move the valuables in, and I stand by it. The gap I hadn’t accounted for is the one between a posture you can design and a posture a fresh account will actually run on its first day.

SubscriptionRequiredException

GuardDuty and Security Hub are first-party AWS services, but they are not on by default. They have to be activated for the account before anything can configure them. Point OpenTofu at a brand-new account that has never had them switched on and you don’t get a security baseline, you get a SubscriptionRequiredException and a failed apply. The locks you carefully specced won’t fit, because the door hasn’t been registered as a door yet.

I’d actually met this exact wall from another direction: an aws-nuke dry-run on a fresh account throws screenfuls of the same exception for every service you’ve never enabled. Same root cause, different tool. A new AWS account is a quieter, emptier place than the console makes it look.

There’s a second, duller reason too, and it belongs in the open: GuardDuty and Security Hub both cost money once their free trial lapses, and the budget for this account wasn’t in place yet. Enabling a service you can’t yet afford to keep on is its own small mistake.

Switching two watchers back off, on purpose

So I did the thing that felt wrong and was right. I switched two of them off, and wrote the reason straight into the module call:

# GuardDuty + Security Hub are deferred. The account's first-time
# service activation is pending (it returns SubscriptionRequiredException)
# and the services carry an ongoing cost beyond their free trial.
# Re-enable by flipping both to true (or deleting these lines) once
# activation clears and the budget is in place.
enable_guardduty = false
enable_securityhub = false

Two things make that a deferral rather than a retreat. First, it’s scoped: Access Analyzer needs no subscription and costs nothing, so it stays on. The account isn’t unwatched, it’s watched by the part that can watch it right now. Second, it carries its own undo. The comment is the re-enable instructions, the toggles sit right there, and flipping them back is a one-line change the day activation clears and the budget lands. A disabled control with a written path back is a deferral. A disabled control with no note is a hole someone finds in a year.

Why this lives in the module, not in a hack

I could only do this cleanly because the module exposes each service as its own toggle, and it only gained that recently; the previous version was all-or-nothing on threat detection. Granular enable_guardduty and enable_securityhub flags are exactly what let you say “this account, these two, not yet” without forking the module or commenting resources out. It’s the difference between a baseline you adapt per account and one you fight.

The honest version of “secure by default”

The hardening post wasn’t wrong. It was idealised. Secure-by-default is the right aim, and on a mature account the baseline goes on clean. On a brand-new one, reality intrudes: services that must be activated before they can be configured, costs that need a budget before they can be incurred. The honest response isn’t to pretend the posture is fully live when it isn’t. It’s to enable what the account will take, defer what it won’t, write down precisely why and how to finish, and leave Access Analyzer watching in the meantime. GuardDuty will be looking soon enough. It just wasn’t looking on day one, and saying so is better than a comment-free enable = true that quietly errored on every apply.

A 403 you can't fix in IAM

Thu, 14 May 2026 00:00:00 +0000

The OIDC post explained the handshake that lets a GitLab pipeline deploy to AWS with no stored key. This is the story of the first time I got it wrong, and spent an afternoon fixing the wrong thing. The error was a flat 403 from AWS, and the maddening part is that no amount of editing the IAM policy was ever going to fix it.

A 403 on the first real run

The OIDC post covered the handshake: GitLab CI mints a signed token, AWS exchanges it for short-lived credentials against a role whose trust policy names the pipeline. During the GitLab migration I wired exactly that up for the infra repo, including a trust policy condition meant to let merge-request pipelines run a plan.

The first merge request that should have triggered tofu-plan didn’t run it. The job failed, and the error from AWS was a flat AccessDenied. A 403.

The instinct, and why it wastes an afternoon

The instinct on an IAM 403 is immediate and almost always right: the policy’s wrong, so go and edit the policy. Tighten the condition. Loosen the condition. Check the wildcard. Re-read the sub pattern character by character.

All of that was wasted, and it was wasted for a reason that took me far too long to see. The trust policy wasn’t matching the wrong value. It was matching a value that does not exist. No amount of editing a condition makes it match a thing that’s never present.

What is actually in the token

GitLab’s OIDC token has a sub claim that encodes the pipeline’s context, and part of that encoding is a ref_type. I’d assumed ref_type could be branch, tag, or mr, because a pipeline can certainly be a branch pipeline, a tag pipeline, or a merge-request pipeline. So the trust policy, for the plan job, matched a sub containing ref_type:mr.

That assumption was wrong. GitLab’s ref_type is branch or tag. That’s the entire set. There is no mr.

A merge-request pipeline doesn’t run against a merge-request ref. It runs against the source branch. So its token’s sub carries ref_type:branch, like any other branch pipeline. The trust policy condition asked for ref_type:mr, GitLab never puts mr in a token, the condition was therefore never true, and every merge-request pipeline got a 403. Forever, until the policy stopped asking for a claim that isn’t real.

The fix, and the lesson worth more than the fix

The fix is small once it’s visible: match ref_type:branch and narrow it down by branch name or project path instead. An afternoon of policy edits, and the actual change is one word.

The lesson is the part worth keeping. When an OIDC trust fails, the useful question is never “is my policy clever enough”. It’s “what’s actually in the token”. An OIDC trust policy can only ever match the claims the identity provider genuinely asserts, and the gap between what a provider asserts and what you assumed it asserts is precisely where this class of bug lives.

So the move, when an OIDC handshake 403s, is to get hold of a real token and decode it. Look at the actual sub, the actual claims, the actual values. Match what’s there. A 403 that survives every sensible edit to the policy is usually not a policy that’s too loose or too strict. It’s a policy matching a claim that was never going to be in the token.

The habit it left behind

I wired an OIDC trust policy to let merge-request pipelines plan, by matching a sub claim with ref_type:mr. The first real merge request got a 403, and no edit to the policy fixed it, because GitLab’s ref_type is only ever branch or tag. A merge-request pipeline runs on a branch ref, so the mr value the policy demanded was never in any token.

The fix was one word. The habit it left behind is the valuable bit: when an OIDC trust fails, stop editing the policy and go and read a real token. A trust policy can only match what the provider actually asserts, and “what I assumed it asserts” is where the 403 was hiding the whole time. (If this shape of bug feels familiar by the end of the series, that’s not an accident: I come back to it with two more from exactly the same family.)

Routing security findings without the noise

Tue, 12 May 2026 00:00:00 +0000

Turning on GuardDuty and Security Hub gives you threat detection. It also gives you a firehose. And an alert system that dutifully forwards everything in that firehose isn’t monitoring, it’s a very efficient way of training your team to ignore alerts. So the alerts module’s real job isn’t detection at all. It’s deciding what’s actually worth interrupting a human for, and the interesting part is everything it deliberately throws away.

Detection is the easy half

Switching on threat detection in an AWS account is a few resources. GuardDuty, Security Hub with its standards, IAM Access Analyzer: the security baseline does exactly that. From then on, the account is generating findings.

And it generates a lot of them. Plenty are low-severity, informational, or simply the normal texture of a cloud account. If you wire every finding to an email or a pager, you haven’t built monitoring. You’ve built noise. And noise has a specific failure mode: people stop reading it, and the one finding that genuinely mattered scrolls past unread alongside two hundred that didn’t.

So the valuable work isn’t detection. It’s routing: deciding what’s worth interrupting a human for, and letting the rest sit quietly in a console for whenever someone reviews it.

Forward the severe, leave the rest

The alerts module routes findings with EventBridge rules into an SNS topic that emails out. The rules are deliberately picky. GuardDuty findings are forwarded only at severity 7 and above. Security Hub findings are forwarded only at HIGH and CRITICAL.

Everything below those thresholds isn’t discarded. It’s still in GuardDuty and Security Hub, where someone doing a review will see it. It just doesn’t get to interrupt anyone’s day. The threshold is the line between “look at this now” and “look at this sometime”.

The duplicate you would otherwise send twice

Here’s the subtle one, and it’s the kind of thing you only find by looking closely at where findings come from.

Security Hub is an aggregator. It pulls findings in from other services, GuardDuty among them. So a single GuardDuty finding can show up in two places: in GuardDuty itself, and again in Security Hub as an aggregated copy.

A rule on GuardDuty findings and a rule on Security Hub HIGH/CRITICAL findings would therefore both fire for the same underlying GuardDuty finding. One event, two emails. Do that across an account and a meaningful fraction of your alert volume is just the same findings counted twice, which is its own kind of noise.

So the Security Hub rule explicitly excludes findings whose ProductName is GuardDuty, with an anything-but match. GuardDuty findings come through the GuardDuty rule. The Security Hub rule handles everything Security Hub adds that GuardDuty didn’t already report. One finding, one alert, regardless of how many services it passed through.

Two tripwires on the root account

Findings are about threats the detectors recognise. The module adds two alarms about something simpler: the root account doing anything at all.

One CloudWatch alarm fires on a root console sign-in. The other fires on any root API call that isn’t a console login. In a well-run AWS account, the root user does almost nothing after initial setup: day-to-day work happens through roles. So root activity isn’t a “finding” to be assessed for severity. It’s a tripwire. Any of it, in an account that should be silent, is worth an immediate look, and the two alarms say so directly.

Why a quiet alert stream matters here

This is monitoring for the account that’s going to hold the release-signing key, and that raises the stakes on getting the routing right.

If a key-bearing account ever does come under attack, the alert that says so has to be seen. An alert stream that’s mostly noise and duplicates is, functionally, no alerting at all, because the people who’d act on it have long since tuned it out. Routing the stream down to “severe, deduplicated, plus root tripwires” is what keeps it something a human will still read on the day it finally matters.

The short version

GuardDuty and Security Hub make detection easy. The hard, valuable part is routing: forwarding what deserves to interrupt someone and leaving the rest in a console.

The alerts module forwards GuardDuty at severity 7-plus and Security Hub at HIGH/CRITICAL, and it drops the duplicate that aggregation creates by excluding GuardDuty-sourced findings from the Security Hub rule, so one finding is one alert. Two CloudWatch alarms act as tripwires on root-account activity, which should be near-zero. For the account that will hold the signing key, a quiet, trustworthy alert stream isn’t a nicety. It’s the difference between monitoring and theatre.

Hardening the account that will hold the keys

Sat, 09 May 2026 00:00:00 +0000

Bootstrapping the account got it ready: somewhere to store state, an identity to deploy as, enough for the next tofu apply to run. Ready is not the same as safe. An account with no audit trail, nothing watching it, and no considered way for a human to get in is fine for experimenting and absolutely not where you put the most sensitive key in the system. So before the signing key goes anywhere near it, the account gets a security baseline.

Ready is not the same as safe

The bootstrap post ended with an account that was ready: it had somewhere to store state and a CI identity to deploy as. The next tofu apply could run.

Ready is not safe. That account still has no audit trail, so nobody could tell you afterwards what happened in it. It has no threat detection, so nothing is watching. Its defaults are AWS’s defaults, which are not a security posture. There’s no considered way for a human to get in. An account in that condition is fine for experimenting. It’s not somewhere you put the most sensitive key in the whole system.

So before the signing key is anywhere near it, the account gets a security baseline.

The baseline, in one downstream stack

terraform-aws-security-baseline is that baseline, and it’s exactly the downstream stack the bootstrap post promised: applied through the automation role bootstrap created, not bootstrapped specially.

It’s six sub-modules, each behind an enable_* toggle: account-hardening (IAM password policy, account-wide S3 public-access blocking, default EBS encryption), audit-logging (a multi-region CloudTrail with log-file validation), aws-config, threat-detection (GuardDuty, Security Hub, IAM Access Analyzer), alerts, and operator-role. Together they turn a bare account into one that records what happens, watches for trouble, and controls who gets in.

Most of those are the expected baseline. The operator role is the one worth slowing down on, because it’s built backwards from how people usually think about an admin role.

The operator role, and the inversion

InfraAdmin is the human way into the account: the role a person assumes to do operator work. Two things define it.

The trust policy decides who may assume it. It trusts only the account root principal, and it requires multi-factor authentication: the assume call must carry aws:MultiFactorAuthPresent, and aws:MultiFactorAuthAge bounds how recently that MFA was performed. No MFA, no role. So far this is a careful but ordinary admin role.

The inversion is a second, separate inline policy, and it’s almost entirely Deny. It denies, using NotAction, anything where aws:RequestedRegion falls outside an allowed set of regions. The role’s power comes from an admin grant. This inline policy fences that power.

That’s the part worth holding onto. People picture an admin role as a list of what it can do. This one is better understood by what it cannot: it cannot act outside its permitted regions, full stop. A fat-fingered command, or a compromised session, cannot quietly spin resources up in some region nobody’s watching. The fence is as much the point of the role as the grant is.

The carve-out, because honesty

There’s a fiddly detail, and it’s the kind of thing that makes the region fence real rather than theoretical.

Some AWS services are global. IAM, CloudFront, Route 53 and friends have no region, and they don’t honour aws:RequestedRegion. A naive region-deny would therefore deny calls to IAM, and you’d lock yourself out of the very service you manage access with. (A close cousin of the kind of self-inflicted lockout I’ll come back to in a later post.)

So the Deny carries explicit carve-outs for the global services. It isn’t elegant, and it can’t be: the global-versus-regional split is just a fact of AWS, and a correct region fence has to account for it. The carve-out list is the real cost of the control working.

Harden the room, then move the keys in

There’s an order to all of this, and the order is the argument.

The account that will hold the signing key has to be audited before the key arrives, so that from day one every call against it is in CloudTrail. It has to be watched before the key arrives, so GuardDuty is already looking. It has to be access-controlled before the key arrives, so the only human path in is MFA-gated and region-fenced.

You don’t move something valuable into a room and then think about locks. You build the room, fit the locks, check they work, and then move the valuable thing in. The security baseline is fitting the locks. The signing key comes later, into a room already built for it.

Worth remembering

Bootstrapping an account makes it ready for the next deploy. It does not make it safe to hold anything that matters. terraform-aws-security-baseline is the downstream stack that closes that gap: audit logging, AWS Config, threat detection, account hardening, and an operator role, applied through the CI role bootstrap created.

The operator role is the piece to study. It’s MFA-gated on the way in, and then fenced by a separate, almost-all-Deny inline policy that confines it to permitted regions, with carve-outs for the global services that have no region. An admin role defined as much by its fence as its grant. Harden the room first; the keys move in afterwards.

No access keys in CI

Fri, 08 May 2026 00:00:00 +0000

A long-lived AWS access key, sitting in a CI system, is just about the single credential I’d most like to be rid of. It’s powerful, it never expires unless someone remembers to rotate it (nobody remembers to rotate it), and it lives in one of the most attractive targets in the whole supply chain. For infrastructure that’s eventually going to hold a release-signing key, it’s exactly the wrong place to start. So the phpboyscout infrastructure has no AWS access key in CI at all. None.

The access key you don’t want

A CI pipeline that runs tofu apply against AWS needs AWS credentials. The traditional way to give it some is an IAM user with an access key pair, pasted into the CI system as a masked variable.

Look at what that key is. It’s long-lived: it works until someone remembers to rotate it, and rotating it is a chore, so mostly nobody does. It’s powerful: it can apply infrastructure, so it can do nearly anything. And it’s sitting in a CI system, which is one of the most attractive targets in your whole supply chain. You’ve taken your highest-value credential and stored a permanent copy of it in a place built for running automated jobs.

For infrastructure that’s going to hold a release-signing key, that’s precisely the wrong starting point. So the phpboyscout infrastructure has no AWS access key in CI at all. Not a well-guarded one. None.

Federation instead of a stored secret

The replacement is OIDC federation, and the shape of it is worth walking through, because it’s genuinely different from “a secret, but better”.

A modern CI platform can mint an OIDC token. GitLab does this with an id_tokens: block: at job time, GitLab issues a short-lived JSON Web Token, signed by GitLab, that asserts a set of facts. This is project X. This is pipeline Y. This is running on ref Z, of this type.

AWS can consume that. The sts:AssumeRoleWithWebIdentity call takes such a token and, if it satisfies an IAM role’s trust policy, returns short-lived AWS credentials for that role. The trust policy is where the control lives: it names GitLab as a trusted token issuer, and it constrains the token’s sub claim so that only the specific project, and the specific refs, you intend can assume the role.

Put it together: the pipeline asks GitLab for a token, hands it to AWS, and gets back credentials that last about an hour and are scoped to one role. Nothing long-lived is stored anywhere. The credential exists only for the job that needs it, and it can’t be stolen from a CI variable store, because it was never in one.

Two halves of one handshake

That handshake is built by two of the repos in this series, each owning one side.

terraform-aws-bootstrap builds the AWS half, in its automation-iam module: it registers GitLab as an OIDC identity provider in the account, and it creates the automation role with the trust policy that decides which pipelines may assume it.

The CI components build the consuming half: the id_tokens: block that asks GitLab for the JWT, and then simply letting the AWS provider’s own credential chain perform the exchange. The pipeline doesn’t call sts by hand. It presents the token; the SDK does the rest.

The gotcha: don’t set a profile

There’s one quiet way to break this, and a stack can look completely correct while doing it.

The AWS SDK finds credentials by walking a chain of sources in order. The web-identity path, the one that uses the OIDC token, is one link in that chain. It triggers off environment variables the CI sets up automatically.

But if the aws provider block has a hardcoded profile = "...", the SDK takes the profile link of the chain instead, and never reaches the web-identity link. A profile line is the sort of thing that ends up in a provider block from someone’s local development setup, where it’s exactly right. Committed and run in CI, it silently short-circuits the federation. The pipeline either fails to find credentials, or finds the wrong ones.

The rule is simple once you know it: the provider block that runs in CI must not name a profile. Leave the chain free to find the web identity. It’s the kind of bug that teaches you to be precise about which link of the credential chain you’re actually relying on.

The bottom line

Giving CI an AWS access key means storing your most powerful, longest-lived credential in one of your most exposed systems. OIDC federation removes it entirely. The CI platform mints a short-lived signed token, AWS exchanges it via AssumeRoleWithWebIdentity for hour-long credentials against a role whose trust policy names the exact pipeline, and nothing permanent is stored.

terraform-aws-bootstrap builds the AWS side, the identity provider and the trust policy; the CI components build the consuming side, the token request. The one trap is a hardcoded profile in the provider block, which short-circuits the SDK’s credential chain before it reaches the web-identity path. Get that right, and a pipeline deploys to AWS as a verifiable, short-lived identity, with no key to steal.

Two layers of tags, and which one wins

Fri, 08 May 2026 00:00:00 +0000

Tagging cloud resources is one of those jobs that’s trivial to do badly and surprisingly fiddly to do well. Everyone agrees resources should be tagged. The argument nobody quite has out loud is where the tags should come from, and getting that wrong gives you either a giant copy-pasted tag block on every resource, or a set of tags that quietly disagree with each other across the account.

Tags answer two different questions

If you look at what tags are actually for, they split cleanly into two kinds, and the split is the whole point.

Some tags are true of every single resource in the account, identically. The environment it belongs to. The fact that OpenTofu manages it. The project or owner it rolls up to for cost reporting. These are invariants: a resource that didn’t carry them would be the bug.

Other tags are specific to a particular piece of infrastructure. Which component this resource belongs to, what subsystem it’s part of. The CloudTrail bucket is part of audit logging; the Config recorder is part of aws-config. That’s a fact about the module, not about the account.

Treat those two kinds the same and you end up repeating the invariants by hand on every resource, which is exactly the copy-paste that drifts. So the infra setup gives each kind its own home.

Layer one: declared once, on the provider

The invariants live on the AWS provider itself, as default_tags, set one time in the provider block:

provider "aws" {
 # ...
 default_tags {
 tags = {
 # Environment, project, managed-by: the things true of
 # every resource in this account.
 }
 }
}

default_tags applies those tags to every taggable resource the provider creates, automatically, without a single resource having to mention them. Change the environment label once, here, and it propagates to everything on the next apply. No resource carries a copy; they all inherit the originals. The invariants are stated exactly once, in the one place that’s true for all of them.

Layer two: merged in by the module

The resource-specific tags live where the resource does: inside the module. Each module merges its own component tag over whatever tags it was handed, which you can see in the public terraform-aws-security-baseline modules:

tags = merge({ Component = "aws-config" }, var.tags)

So the aws-config module stamps Component = "aws-config" onto the things it builds, the account-hardening module stamps its own, and so on. The caller can pass extra tags down through var.tags, and because they come last in the merge, the caller can override the module’s defaults when it genuinely needs to. Module-specific knowledge stays in the module; per-call adjustments stay with the caller.

Which layer wins

Now the question that actually bites: a resource is getting tags from the provider’s default_tags and from the module’s merge. What happens when both set the same key?

The resource-level tags win. AWS’s provider treats tags set directly on a resource as an override of default_tags on a key collision, so the module’s merged tags take precedence over the account-wide defaults. That’s the right way round: the invariants are a sensible baseline, and a module that has a specific reason to set a key differently can, without having to reach up and edit the provider block that everything else depends on. Most of the time the two layers are simply disjoint, the invariants saying what account this is and the module tags saying what this resource is for, and they never collide at all. When they do, local intent beats the global default, which is the precedence you’d want.

Why bother splitting it

The payoff is that neither layer has to know about the other. The provider declares the invariants once and never thinks about components. Each module declares its component and never hard-codes the environment. Add a new module and it inherits every account-wide tag for free, while contributing its own. Change an account-wide tag and you touch one block, not two hundred resources. The tags stay consistent not because someone’s policing them, but because the place each tag is declared is the one place it can be declared.

The short version

Resource tags answer two questions, and they want two homes. Account-wide invariants (environment, ownership, managed-by) go on the provider’s default_tags, declared once and inherited by everything. Resource-specific tags go in the module, via merge({ Component = "..." }, var.tags), so each module owns its own labels and the caller can still override. On a key conflict the resource-level tag wins, which means the module’s intent beats the account default exactly when it should. Two layers, each declared in the one place it belongs, and no copy-pasted tag block anywhere in sight.

The chicken-and-egg of remote state

Wed, 06 May 2026 00:00:00 +0000

Here’s a puzzle that every infrastructure-as-code setup hits exactly once, right at the very beginning, and then never again. An OpenTofu stack stores its state in a backend. The bootstrap stack I wrote about last time has a particular job, and part of that job is to create the backend that remote state lives in. So where does the bootstrap stack store its own state, on the very first run, before it’s built the place state is supposed to go?

Where does the state of the thing that makes the state store live?

That’s the puzzle, and it’s a real ordering deadlock rather than a riddle.

An OpenTofu stack keeps a state file, and for anything shared that state file lives in a remote backend: on AWS, an S3 bucket. Fine. But the bootstrap stack has a particular job, and part of that job is to create the S3 bucket that remote state lives in.

So walk through the first run. Bootstrap has never been applied. The state bucket doesn’t exist, because creating it is what bootstrap is for. Bootstrap needs somewhere to store its own state. The only place that would make sense is the bucket it’s about to create, which isn’t there yet. The thing that builds the state store can’t store its state in the state store.

Run local, then migrate

The way out is a two-step that OpenTofu supports directly.

Bootstrap starts configured with a local backend: backend "local" {}. State is just a file on the operator’s machine. With that in place, the first tofu apply runs. It creates the S3 bucket and the KMS key, and records all of it in the local state file.

Now the bucket exists. So the backend configuration is rewritten to point at it: an s3 backend block naming the new bucket. Then tofu init -migrate-state. OpenTofu sees the backend has changed, picks up the local state file, and copies it into the S3 bucket. From that point on, bootstrap’s own state lives in the bucket that bootstrap created. The egg has laid the chicken.

The local backend was a scaffold. It existed for exactly one apply, to break the ordering deadlock, and then the state moved off it and it was never used again.

It happened twice

The infra repo actually did this migration twice, and the second time is the proof that the pattern is general rather than a one-off trick.

The first migration was the one above: local to S3, at the very start. The second came later, during the move from GitHub to GitLab. GitLab offers a managed HTTP state backend, and infra chose to use it. So the backend block was rewritten again, this time from s3 to http, and tofu init -migrate-state ran again, copying the state from the S3 bucket to GitLab’s backend.

The same move, twice, against three different backends. That’s the useful lesson hiding in the chicken-and-egg story. State is portable. The backend is just where you currently keep it, not a property of the stack itself, and moving it is a routine, supported operation rather than surgery.

Why this is the honest answer, not a hack

It’s easy to look at “apply once with a local backend, then migrate” and feel it’s a bit of a smell, a workaround for something that should have been cleaner.

It isn’t. It’s the honest answer to a real ordering problem, and the alternatives are worse.

The obvious alternative is to create the state bucket by hand, in the console, before running bootstrap at all. But then the most important bucket in the account is unmanaged. It exists outside every OpenTofu graph, nobody’s code describes it, its encryption and policy and prevent_destroy are whatever someone clicked that day, and it drifts. The local-then-migrate dance avoids exactly that. The bucket is created by bootstrap, described in code, and tracked in bootstrap’s own state from its very first apply. It’s managed from birth.

The chicken-and-egg isn’t a flaw to be embarrassed about. It’s just the shape of the problem when a stack has to build its own foundations, and OpenTofu’s -migrate-state is the supported tool for exactly that shape.

Pulling it together

Every OpenTofu stack needs a backend to store state, and the bootstrap stack’s job is to create the backend, so on its first run the bucket it needs doesn’t yet exist.

The resolution is to run bootstrap once with a local backend, let that apply create the bucket and key, then rewrite the backend configuration and tofu init -migrate-state the state into the bucket bootstrap just made. The infra repo did it twice, local to S3 and later S3 to GitLab, which shows the real point: state is portable, and the backend is just where you keep it. Doing it this way, rather than hand-creating the bucket, is what keeps that critical bucket managed in code from its very first day.

The cleanup tool that almost deleted its own hands

Tue, 05 May 2026 00:00:00 +0000

The first time I pointed aws-nuke at a real account, the dry-run printed hundreds of lines of angry red text and my stomach dropped. Then I read it properly, and two things turned out to be true at once. Almost all of that red was noise. And the one operation I genuinely should have worried about wasn’t red at all.

A tool whose whole job is destruction

aws-nuke deletes everything in an AWS account. That’s the point of it: when you spin up a throwaway account to try something, aws-nuke is how you tear it back down to nothing afterwards rather than leaving resources quietly billing forever. go-tool-base’s bootstrap renders a scoped aws-nuke config for exactly this, from a nuke-config module, so the teardown is described in code rather than typed by hand at the worst possible moment.

A tool that deletes everything is a tool you run in dry-run first, every single time, and read the output before you let it touch anything. So that’s what I did. And the output was alarming in a way that turned out to be completely meaningless, and reassuring in a way that turned out to hide the one real hazard.

The wall of red that means nothing

A fresh account threw up screen after screen of SubscriptionRequiredException. Hundreds of lines, all red, all looking like something had gone badly wrong.

They hadn’t. aws-nuke works by asking every region “do you have any of this kind of resource?”, for every kind of resource it knows about. On a brand-new account you’ve never enabled most services in most regions, so the API’s honest answer is “you’re not subscribed to that here”, which surfaces as an exception, which the tool dutifully logs in red. It isn’t a failure. It’s the sound of an empty account being asked four hundred questions and answering “nothing here” to almost all of them.

The skill, and it is a skill, is learning to read a destructive tool’s dry-run and tell the noise from the signal. SubscriptionRequiredException on a fresh account is noise. Once you know that, the wall of red stops being frightening and becomes scenery.

There’s a related trap in the same neighbourhood, and the nuke-config module’s own regions variable documents it. aws-nuke has a special global pseudo-region for things that don’t live in one place (IAM, Route 53, CloudFront), and then the actual regions for everything else. It also accepts all, meaning every enabled region. Mixing all with explicit region values scans some regions twice and muddies the output, so the module’s guidance is to pick one approach or the other. More scenery you have to learn to read before the genuinely important line will stand out.

The line that should have scared me, and didn’t look like it

Buried in that calm-looking eye of the storm, among the resources aws-nuke intended to delete, were the IAM resources granting the identity running the nuke its administrative access.

Sit with that for a second. aws-nuke runs as some principal with enough power to delete everything. To delete EVERYTHING, it has to delete IAM resources too. And if the plan deletes the very grant that gives the running identity its admin before it’s finished, the tool strands itself partway through: no permissions left to complete the teardown, and now you’ve got a half-nuked account and a principal that can’t act on it. The cleanup tool sawing off the branch it’s standing on, calmly, without a single red line to warn you, because from the API’s point of view deleting that resource is a perfectly valid request.

That’s the operation that actually mattered, and it was the quietest thing in the output.

Two ways to keep its hands attached

The fix has two halves, one explicit and one structural.

The explicit half is to preserve the privilege path. The nuke-config module passes a caller-supplied set of filters straight through into the rendered config, so you tell aws-nuke “everything except these”. You exclude the identity running the nuke, and the policy and path that grant it admin, from deletion. The tool cleans the account and leaves its own hands alone, because you told it which resources are off-limits.

The structural half is to not give it a tempting separate thing to delete in the first place. If an identity gets its admin through an IAM group it belongs to, that group is its own deletable resource, one more thing in the plan, one more way to be stranded. The automation role in terraform-aws-bootstrap instead takes its policies as direct attachments to the role itself:

resource "aws_iam_role_policy_attachment" "gitlab" {
 for_each = local.is_gitlab ? var.policy_arns : {}
 role = aws_iam_role.this.name
 policy_arn = each.value
}

No intermediary group sitting there as a separate, deletable object. Flattening the privilege onto the role makes the dependency simpler to reason about and gives the cleanup tool one fewer foot-gun to find. Belt and braces: filter the path out explicitly, and don’t build a structure that invites the problem.

What it comes down to

A destructive tool’s dry-run is the most valuable thing it produces, and reading it well is its own competence. On a fresh account, the screenfuls of red SubscriptionRequiredException are noise, the sound of an empty account answering “nothing here”, and the all-versus-global region wrinkle is more of the same. Learn to see past all of it, because the operation that can actually hurt you is rarely the one shouting. Mine was the calm, unremarkable line proposing to delete the admin grant the nuke needed to finish its own job.

Keep the cleanup tool’s hands attached: filter the privileged path out of the teardown so it’s never a candidate for deletion, and attach that privilege directly rather than through a group that’s just one more thing to delete. Then let it loose on everything else, which is, after all, what you brought it in to do.

The security finding you must not fix

Mon, 04 May 2026 00:00:00 +0000

A security scanner flagged a finding in my Terraform, and the correct response, the one I had to talk myself into, was to leave it exactly as it was. Not because the finding was wrong about what the code did. It was right. It’s that doing what it asked would have quietly bricked the account.

A finding that looks open and shut

I run checkov over the infrastructure as part of CI, and on the KMS key that protects the Terraform state bucket it raised CKV_AWS_111: a key policy that grants kms:* is overly permissive. On the face of it, unarguable. The policy says the account root can perform any KMS action on the key, with a resource of *. A wildcard action and a wildcard resource is the exact shape a scanner is built to shout about, and ninety-nine times in a hundred it’d be right to.

This was the hundredth time.

Why narrowing it bricks the key

Here’s the bit that turns the finding on its head. That kms:*-for-root statement isn’t an over-broad grant I left lying around. It’s the default key policy AWS itself applies, and it’s load-bearing in a way that’s easy to miss.

A KMS key is administered through its own key policy, and that policy is the only way in. Unlike most resources, IAM permissions elsewhere can’t grant access to a key whose policy doesn’t allow it. So if you “tighten” the key by removing root’s full control, and you don’t perfectly replace it with some other administrative principal, you can end up with a key that nobody can administer. Not you, not root, not a future you with a very good reason. KMS will not let you recover it. The key, and anything it encrypts, is stranded.

The kms:*-for-root statement is what keeps the account’s own root able to manage the key as a last resort. It’s not the vulnerability. It’s the escape hatch, and the scanner was asking me to weld it shut.

So the finding gets suppressed, out loud

The answer isn’t to silence the scanner globally, and it isn’t to obey it. It’s to suppress this finding, on this resource, with the reasoning written right there next to it, from modules/state-backend/main.tf:

data "aws_iam_policy_document" "kms" {
 # checkov:skip=CKV_AWS_111:kms:* on the CMK for the account root is the
 # AWS-documented pattern; narrowing it risks an unrecoverable lockout from
 # the key. See https://docs.aws.amazon.com/kms/latest/developerguide/key-policy-default.html
 statement {
 sid = "AllowAccountRootAdmin"
 effect = "Allow"
 principals {
 type = "AWS"
 identifiers = ["arn:aws:iam::${var.account_id}:root"]
 }
 actions = ["kms:*"]
 resources = ["*"]
 }
}

The skip carries a reason and a link to AWS’s own documentation of why this is the recommended default. That matters more than it looks. A bare # checkov:skip with no explanation is indistinguishable from laziness, and the next person to read it (quite possibly me, a year on) has no way to tell whether it was a considered decision or someone making a red mark go away. A skip with a documented reason is a decision you can audit. The finding is still visible in the sense that the suppression is right there in the code, attached to the thing it’s about, defensible out loud.

A scanner is an argument, not an order

The wider lesson is the one worth keeping, because it generalises well past this one key. A static-analysis finding is a prompt to think, not an instruction to comply with. Most of the time thinking leads you straight to “yes, fix it”, and you should. But a scanner encodes a general rule, and general rules meet specific contexts where they’re wrong, or merely irrelevant, or, in the rare and dangerous case, actively harmful to obey. kms:* for root on a customer-managed key is that last kind: the tool’s general rule (“wildcards are bad”) collides with a hard AWS-specific fact (“root must retain control of the key or it’s gone”).

The discipline that keeps this honest is the one in the code above. You don’t get to ignore a finding. You get to suppress it, scoped to the exact resource, with a reason a reviewer can weigh. Cheap enough that you’ll do it properly, costly enough that you won’t paper over a real finding by reflex.

The bottom line

checkov was right that the state-bucket key grants the account root kms:* on *. It was wrong that I should narrow it, because that statement is AWS’s documented default and the thing that stops the key becoming permanently unadministrable. The fix was a scoped checkov:skip carrying its reasoning and a link to the AWS docs, so the decision lives next to the code and can be defended rather than merely trusted.

Treat your scanner as a sharp, tireless colleague who’s usually right and occasionally, confidently, about to lock you out of your own key. Read every finding. Obey most of them. And write down, in the open, the rare one you mustn’t.

A state bucket that defends itself

Sat, 02 May 2026 00:00:00 +0000

OpenTofu’s remote state file is, quietly, the most sensitive thing in an infrastructure repo. It’s a plain JSON document listing every resource you manage, every ID, and, depending on your providers, the odd secret in clear text. So the S3 bucket that holds it can’t just be a bucket. It has to actively defend itself, on three separate fronts.

The most sensitive file in the repo

OpenTofu, like Terraform, keeps a state file: a JSON document recording every resource the stack manages, its real-world ID, and its attributes. It’s how the tool knows what already exists. It’s also, quietly, the most sensitive file in the whole repo. It can hold resource identifiers an attacker would value, and depending on the providers in play it can hold secret values in clear text.

Three bad things can happen to it. It can be deleted, and now the tool has forgotten everything it manages. It can be read by someone who shouldn’t. It can be corrupted by two runs writing at once. The bucket that holds remote state has to defend against all three, and terraform-aws-bootstrap’s state-backend module is built around doing exactly that.

The DynamoDB lock table is gone

Start with the corruption problem, because the answer changed recently.

The long-standing pattern for remote state on AWS was an S3 bucket plus a DynamoDB table. S3 held the state; the DynamoDB table held a lock, so two apply runs couldn’t write at once. Everyone who’s done Terraform on AWS has provisioned that table, probably more times than they’d care to count.

OpenTofu 1.10 made it unnecessary. The S3 backend gained use_lockfile, which does the locking with a small lock object in the same bucket, using S3’s conditional-write support. No separate table. The state backend is now genuinely one bucket and one key, with the lock living beside the state. It’s one fewer resource to create, one fewer thing to pay for, and one fewer moving part to reason about. The module takes the new path, and the DynamoDB table simply isn’t there.

A bucket you can’t delete by accident

Deletion is guarded with lifecycle { prevent_destroy = true } on the bucket. With that set, OpenTofu refuses to produce a plan that would destroy the bucket. A stray tofu destroy, a refactor that drops the resource, an accidental rename: all of them fail loudly instead of quietly taking the state bucket with them.

This is also why the state-backend module is hand-rolled from raw aws_s3_bucket resources rather than wrapping a community module like terraform-aws-modules/s3-bucket. prevent_destroy has to sit on the actual resource, and a lifecycle block isn’t something you can pass into a wrapper module as an input. Hand-rolling the bucket keeps prevent_destroy somewhere you can put it and, just as importantly, somewhere the next reader can see it. (There’s a whole post coming on why I hand-rolled every module; this is one of the reasons in miniature.)

Reject anything encrypted wrong

Confidentiality is the subtle one, because the obvious control isn’t enough.

The bucket has a default encryption configuration: server-side encryption with the customer-managed KMS key. But default encryption is a default. A client making a PutObject call can override it per request, asking for plain AES256 or a different KMS key, and S3 will honour the override.

So the module doesn’t rely on the default. The bucket policy explicitly denies the upload it doesn’t want. It denies any request not over TLS. It denies any PutObject that isn’t using SSE-KMS. And it denies any PutObject that names the wrong KMS key. The default encryption config says “this is what you get if you don’t ask”; the bucket policy says “and you’re not allowed to ask for anything else”. State can only ever land encrypted, in transit and at rest, under the one key the module controls.

One small companion setting: bucket_key_enabled. With per-object SSE-KMS, every object operation is also a KMS API call, which costs money and can throttle. An S3 Bucket Key collapses those into far fewer KMS calls, cutting per-object KMS traffic by well over ninety per cent. It’s a one-line setting the module turns on and most people forget exists.

In short

Remote state is the most sensitive file an infrastructure repo has, and the bucket that holds it has to defend against deletion, disclosure and corruption.

terraform-aws-bootstrap’s state backend handles corruption with OpenTofu 1.10’s use_lockfile, dropping the old DynamoDB lock table entirely. It guards deletion with prevent_destroy, which is also why the bucket is hand-rolled rather than wrapped. And it guards confidentiality with a bucket policy that denies non-TLS traffic and denies any upload not encrypted with the right KMS key, because default encryption is only a default and a client can override it. The state bucket isn’t just a place to put state. It’s built to refuse every wrong thing that could happen to it.

The bootstrap that does almost nothing

Fri, 01 May 2026 00:00:00 +0000

A brand-new AWS account is a slightly nerve-wracking thing. It can do almost anything, it’s hardened against almost nothing, and the list of stuff you ought to set up before you trust it with anything real is long. The natural instinct is to write one big “set up the account” module that does the whole list in a single apply. I want to talk you out of that, because the bootstrap module I’m happiest with does almost nothing, on purpose.

The first-apply problem

A brand-new AWS account is not ready for anything serious. Before you’d responsibly run real infrastructure into it, you want an account baseline: a password policy, account-wide S3 public-access blocking, default EBS encryption, CloudTrail, AWS Config, GuardDuty, alerting, a sensible human operator role. It’s a long list, and all of it matters.

The instinct, faced with that list, is to write one big “set up the account” module and have it do everything. One tofu apply, a fully prepared account, done.

That instinct is worth resisting, and terraform-aws-bootstrap resists it deliberately.

Three things, and a hard line

terraform-aws-bootstrap does three things:

state-backend, an S3 bucket and a customer-managed KMS key to hold remote Terraform state.
automation-iam, an OIDC identity provider and an IAM role that CI assumes to apply everything else.
nuke-config, which renders an aws-nuke configuration scoped to the account, for tearing a throwaway account back down.

That’s the whole module. Account hardening, CloudTrail, AWS Config, GuardDuty, the operator role, the alerting: none of it is in here. And it’s not absent by accident. The README has a section headed “what’s deliberately NOT in scope” that lists those exclusions out loud. The boundary is written down, because the boundary is the design.

Why the line is exactly there

The reason the line sits where it does is the most useful idea in the module.

Everything bootstrap excludes belongs in a separate stack, applied through the automation role bootstrap creates. Bootstrap’s only job is to get the account to the point where the next tofu apply can run properly: somewhere to store state, and an identity to run as. Once those two things exist, hardening the account isn’t a special bootstrapping act. It’s just another apply, done the normal way: in CI, reviewed, versioned, deployed through the role.

So the account baseline doesn’t need to be bundled into the bootstrap. It needs to be downstream of it. Bootstrap builds the on-ramp; it doesn’t also have to be the motorway.

A narrow module stays re-runnable

There’s a practical payoff to the narrowness, and it’s about fear.

Bootstrap is the one stack that can’t be applied through CI, because it’s what creates the CI identity in the first place. It runs locally, by a human, rarely. That’s exactly the kind of operation you want to be small, boring, and safe to repeat.

A bootstrap module that also did account hardening would be a large, stateful thing managing dozens of resources. Re-running it would be a held-breath operation. Keeping it to three concerns keeps it the opposite: a small stack you can read top to bottom, re-run without anxiety, and reason about completely. The narrowness isn’t minimalism for its own sake. It’s what keeps the one human-applied stack trustworthy.

The boundary is the feature

It’s tempting to judge a module by how much it does. A bootstrap module is the case where that’s exactly backwards. Its value is in how cleanly it stops.

terraform-aws-bootstrap does the bare minimum to make an account ready for the next apply, writes down everything it refuses to do, and hands off to a downstream stack for all of it. The next post follows the trickiest of its three jobs: the state backend has a genuine chicken-and-egg problem, because it has to store Terraform state in a bucket Terraform hasn’t created yet.

Where this leaves us

A fresh AWS account needs a long list of things before it’s safe, and the obvious move is one big module that does the lot. terraform-aws-bootstrap deliberately does only three: a state backend, a CI identity, and an account-scrub config. Everything else is written down as out of scope.

The boundary is the design. The excluded work belongs in a downstream stack applied through the CI role bootstrap creates, so hardening is just a normal reviewed apply rather than a bootstrapping special case. And keeping the one human-run, locally-applied stack small is what keeps it safe to re-run. A bootstrap module is judged by where it stops.