Infrastructure with AWS and OpenTofu

Standing up an AWS account from nothing, with OpenTofu and no long-lived credentials anywhere. The bootstrap that deliberately does almost nothing, and the cleanup tool that nearly deleted its own hands.

14 posts

Start here

What the first stack should and should not do, and the chicken-and-egg problem underneath it.

  1. The bootstrap that does almost nothingAn AWS bootstrap module that does exactly three things, and writes down what it deliberately does not do. Scope as a design decision. Pioneering
  2. The chicken-and-egg of remote stateThe bootstrap stack has to create the bucket its own state lives in. Applying once with a local backend, then migrating the state into it. Pioneering
  3. Why I hand-rolled every moduleWhy security-critical infrastructure got raw resources rather than community wrapper modules, and what that actually cost in lines and in review. Pioneering

State, and keeping it safe

The bucket everything else depends on, and the one graph it all lives in.

  1. A state bucket that defends itselfAn OpenTofu state bucket that defends itself against corruption, deletion and its own operator, using lockfiles and prevent_destroy. Pioneering
  2. One graph, not micro-stacksWhy an infrastructure repo stays one OpenTofu graph rather than splitting into per-concern stacks: the engine enforces ordering for free. Pioneering
  3. Reviewed, then appliedClosing the plan-versus-apply gap in infrastructure CI by applying the saved plan artifact rather than quietly re-planning at apply time. Pioneering
  4. Two layers of tags, and which one winsAWS tagging in two layers: account-wide invariants on the provider default tags, resource specifics in the module, and which one wins on a clash. Pioneering

Getting in without a key

OIDC federation instead of access keys, and the 403 that no IAM change could fix.

  1. No access keys in CIReplacing long-lived AWS access keys in CI with OIDC federation, so the pipeline mints a short-lived token instead of holding a secret. Pioneering
  2. A 403 you can't fix in IAMA GitLab to AWS OIDC trust policy kept returning 403, and no IAM change fixed it: the policy matched a claim GitLab never actually asserts. Pioneering

Hardening, and the bills

The baseline you apply to an account that will hold real keys, and the alarm you want before the surprise.

  1. Hardening the account that will hold the keysApplying a security baseline to a fresh AWS account: audit logging, config recording, threat detection and an operator role that is not root. Pioneering
  2. The security service I had to switch offA fresh AWS account cannot enable GuardDuty or Security Hub without a subscription, so the security baseline failed on the account it protects. Pioneering
  3. Routing security findings without the noiseRouting AWS GuardDuty and Security Hub findings so an alert still means something: forward high severity only, and drop the duplicates. Pioneering
  4. Wire up an AWS billing alarm before the surpriseSetting up an AWS billing alarm before you need one, with a KMS gotcha in the middle that stops the alert ever reaching you. Orienteering
  5. The cleanup tool that almost deleted its own handsReading an aws-nuke dry run: the screenfuls of red are harmless noise, and the real hazard is one quiet line in the middle of them. Pioneering

Where to next

Everything, newest first →