Featured image of post Wire up an AWS billing alarm before the surprise

Wire up an AWS billing alarm before the surprise

Everybody who has run anything on AWS carries the same low-grade dread: the bill you don’t see coming. A cron that spins up an instance and forgets to kill it, a misconfigured NAT gateway billing you by the gigabyte, a test that leaves a fleet running over a bank holiday weekend. The fix is well known and takes ten minutes: set a budget, and have it shout at you when spend crosses a line.

The trap is subtler, and it’s the reason for this post. You can set that budget, wire it to a topic, see the notification sitting in the console exactly as you configured it, feel thoroughly responsible… and then hear nothing at all on the day spend sails past your line. A billing alarm that doesn’t fire is worse than no alarm, because you’ve stopped watching for the thing yourself. So here’s how to wire one up properly, and the one piece AWS will actively nudge you into getting wrong.

The easy part: the budget

A budget is one resource. Give it a monthly cost limit and a threshold to shout at:

resource "aws_budgets_budget" "monthly_cost" {
  name         = "monthly-cost"
  budget_type  = "COST"
  limit_amount = "50"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator       = "GREATER_THAN"
    threshold                 = 80
    threshold_type            = "PERCENTAGE"
    notification_type         = "ACTUAL"
    subscriber_sns_topic_arns = [aws_sns_topic.alerts.arn]
  }
}

You can point a notification straight at an email address, and plenty of people do. I’d send it to an SNS topic instead, because a topic is a fan-out point: today it emails you, tomorrow it also pokes Slack, a Lambda, or your on-call rota, and you change none of the budget wiring to do it. If you’ve built anything security-conscious you probably already have such a topic. The public terraform-aws-security-baseline module, for instance, stands up an alerts topic wired to shout about GuardDuty findings and root-account activity. Hanging your cost alarm off the same wire, so everything that matters arrives in one place, is a tidy instinct.

It’s also exactly where the trap is waiting.

The hard part: encryption locks the alarm out

A well-built alerts topic is encrypted, and it should be. The security-baseline one uses a customer-managed KMS key, so nothing sensitive ever sits in plaintext on the topic. Encrypt the topic, point your budget at it, and nothing about the setup will look wrong. The budget is real, the notification is configured against the topic, terraform apply is happy. But the alarm won’t fire, and AWS is going to be spectacularly unhelpful about why.

Here’s the mechanism, because it’s worth understanding rather than just pasting the fix. Most things that feed an alerts topic arrive through EventBridge, which publishes on your behalf. A budget doesn’t. AWS Budgets publishes to SNS directly, as its own service, under the principal budgets.amazonaws.com. To put a message onto an encrypted topic, whoever’s publishing has to encrypt it first, and that means being allowed to use the topic’s KMS key. EventBridge was granted that. The budget service was not.

You can see the grant the topic already makes in the module’s key policy: EventBridge is named as a user of the key, allowed to generate a data key and decrypt so it can encrypt what it publishes (alerts/main.tf). There’s no equivalent line for budgets.amazonaws.com, because the module was built for security findings, not cost. So the budget goes to publish, can’t touch the key, and the alert never leaves the building.

And here’s the bit that turns a five-minute fix into a lost afternoon: AWS’s own guidance points you the wrong way. Try to attach a budget to an encrypted topic in the console and it stops you with “The SNS topic is encrypted. The SNS topic won’t work without additional permissions. Disable encryption on the topic.” Read that twice. Faced with an encrypted topic a budget can’t use, AWS’s headline suggestion is to turn the encryption off. Don’t. You don’t rip out a good security control to make a billing alert work. And the belated warning you might get when a budget can’t publish, an out-of-band note asking you to make sure budgets is on the topic’s list of allowed publishers, points straight back at the topic’s access policy. Neither message ever mentions the key. Every signal AWS hands you is aimed at the wrong thing, and the real fix is to grant the alert access to the key, exactly the way the module already did for EventBridge.

The two grants that fix it

The budget needs the same treatment EventBridge got, in two places. First, on the KMS key policy, name the budget service as a user of the key (AWS documents this exact requirement):

statement {
  sid    = "AllowBudgetsPublishViaTopic"
  effect = "Allow"
  principals {
    type        = "Service"
    identifiers = ["budgets.amazonaws.com"]
  }
  actions = [
    "kms:GenerateDataKey*",
    "kms:Decrypt",
  ]
  resources = ["*"]
}

Second, on the SNS topic policy, let the budget publish:

statement {
  sid    = "AllowBudgetsPublish"
  effect = "Allow"
  principals {
    type        = "Service"
    identifiers = ["budgets.amazonaws.com"]
  }
  actions   = ["sns:Publish"]
  resources = [aws_sns_topic.alerts.arn]

  condition {
    test     = "StringEquals"
    variable = "aws:SourceAccount"
    values   = [data.aws_caller_identity.current.account_id]
  }
  condition {
    test     = "ArnLike"
    variable = "aws:SourceArn"
    values   = ["arn:aws:budgets::${data.aws_caller_identity.current.account_id}:*"]
  }
}

Those two aws:SourceAccount and aws:SourceArn conditions are AWS’s documented shape for the budget topic policy, and they’re worth keeping: they stop any other account’s budget from ever using your topic (the confused-deputy guard the baseline already applies to its EventBridge statements). The pair of grants above are otherwise twins of what the module writes for EventBridge, with one principal swapped. Add them and the message flows.

One thing that catches people out: this only works with a customer-managed key. If your topic uses the default AWS-managed aws/sns key, you can’t edit its policy to add a service principal, so a budget can never publish to it. That’s a large part of why the security-baseline module brings its own CMK, and it’s the difference between “encrypted” and “encrypted and actually usable by the things that need it”.

Watch it fire once

The most important step is the one people skip, and it’s the whole reason I wrote this down. I won’t believe an alarm works until I’ve watched it fire, and a billing alarm is no exception. Temporarily drop the threshold to something you’re certain you’ve already crossed this month, a single percent if you like, apply, and wait for the message to actually land in your inbox or your Slack. When it does, you know the whole chain is live: budget, topic policy, key policy, subscription. Then put the threshold back.

It takes five minutes and it’s the difference between an alarm and a decoration. The budget was always the easy 80%. The KMS key policy is the fiddly 20% that decides whether, on the day a runaway resource starts eating your account, the thing pages you or just watches.

Built with Hugo
Theme Stack designed by Jimmy