Someone edits a setting, restarts the service, and nothing changes. The file is right. You can cat it back and it says exactly what they typed. The process carries on doing the old thing regardless, and now there’s a call open and four different places that value could be coming from.

I’ve lost afternoons to that, and the annoying part is that it’s never really a mystery. It’s just that nothing in the process can tell you which layer won.

So this walkthrough builds the whole chain a real service uses, defaults under a file under the environment under flags, and then spends most of its time on the more useful half: asking the thing which layer supplied a value, which layers lost, and what would happen if you wrote to it. Twenty minutes, one module, no services and no network past the first go get.

If you want the argument for why config is its own module rather than something the framework owns, that’s a separate post. This one is just using it.

This is the canonical walkthrough from the go/config docs, reproduced here so the two don’t drift. The docs are the source of truth: https://config.go.phpboyscout.uk/tutorials/layering/. I built the module and ran every step against v0.17.2 on Go 1.27.0 before this went out, and every output below is what came back, with one divergence flagged in step 6.

1. Create a module

mkdir cfgdemo && cd cfgdemo
go mod init cfgdemo
go get gitlab.com/phpboyscout/go/config github.com/spf13/pflag

pflag is only for the flags layer in step 5. The other three need nothing beyond the module itself, which is rather the point of it having been extracted.

2. Compile the defaults into the binary

A default isn’t a config file. It ships inside the binary, so the program starts correctly on a machine with no configuration at all, and it can never be written to, because there’s nowhere to persist it. That last bit matters later than you’d think.

main.go:

var builtinDefaults = []byte(`
server:
  host: 0.0.0.0
  port: 8080
  timeout: 30
log:
  level: info
`)

func main() {
	store, err := config.NewStore(context.Background(),
		config.WithReaders(config.NamedSource{
			Name:    "embedded:defaults.yaml",
			Content: builtinDefaults,
		}),
	)
	if err != nil {
		log.Fatal(err)
	}

	v := store.View()
	for _, key := range []string{"server.host", "server.port", "server.timeout", "log.level"} {
		fmt.Printf("%-16s = %-10v %s\n", key, v.Get(key), v.Explain(key))
	}
}

$ go run .
server.host      = 0.0.0.0    server.host = 0.0.0.0 (from default:embedded:defaults.yaml)
server.port      = 8080       server.port = 8080 (from default:embedded:defaults.yaml)
server.timeout   = 30         server.timeout = 30 (from default:embedded:defaults.yaml)
log.level        = info       log.level = info (from default:embedded:defaults.yaml)

Give the source a name you’d recognise at the wrong end of an incident. embedded:defaults.yaml tells you where to go and look; reader1 tells you nothing, and provenance is most of what this module is for.

3. Put a config file over the top

config.yaml, next to main.go. Keep the comments, step 7 comes back to them:

# Deployment settings for the demo service.
server:
  # The port the operators agreed on.
  port: 9090
log:
  level: debug

It sets two of the four keys deliberately. Add it to the store after the defaults:

	fsys, err := config.Dir(".")
	if err != nil {
		log.Fatal(err)
	}

	store, err := config.NewStore(context.Background(),
		config.WithReaders(config.NamedSource{
			Name:    "embedded:defaults.yaml",
			Content: builtinDefaults,
		}),
		config.WithFiles(fsys, "config.yaml"),
	)

$ go run .
server.host      = 0.0.0.0    server.host = 0.0.0.0 (from default:embedded:defaults.yaml)
server.port      = 9090       server.port = 9090 (from config.yaml); also defined in default:embedded:defaults.yaml
server.timeout   = 30         server.timeout = 30 (from default:embedded:defaults.yaml)
log.level        = debug      log.level = debug (from config.yaml); also defined in default:embedded:defaults.yaml

Precedence is the order you added the sources, and later wins. No ranking table to learn, no priority number to set. The file outranks the defaults because WithFiles came after WithReaders, and if you want it the other way round you move the line.

Two keys came from the file, two from the defaults, and the merge happened per key rather than per file. A file that sets one key doesn’t blank out the rest of the tree, which sounds obvious right up until you meet a library that does it the other way.

4. Add the environment

		config.WithEnv("CFGDEMO"),

$ CFGDEMO_SERVER_PORT=7000 go run .
server.port      = 7000       server.port = 7000 (from env:CFGDEMO_SERVER_PORT); also defined in default:embedded:defaults.yaml, config.yaml

CFGDEMO_SERVER_PORT became server.port: prefix stripped, the rest lowercased, underscores into dots.

That prefix isn’t a nicety, it’s a security control. Without one, every variable in the process environment is a candidate configuration key: PATH, HOME, a token injected for something else entirely. The full rules, including what happens when an underscore is ambiguous, are in Environment variables.

5. Add flags on top

Flags go last, because the person typing one has the most immediate intent of anybody involved.

	flags := pflag.NewFlagSet("demo", pflag.ExitOnError)
	flags.Int("server-port", 0, "port to listen on")
	flags.String("log-level", "", "log verbosity")
	if err := flags.Parse(os.Args[1:]); err != nil {
		log.Fatal(err)
	}

and as the last store option:

		config.WithFlags(flags),

$ CFGDEMO_SERVER_PORT=7000 go run . --server-port 6000
server.port      = 6000       server.port = 6000 (from flag:--server-port); also defined in default:embedded:defaults.yaml, config.yaml, env:CFGDEMO_SERVER_PORT

One value, four layers that could have supplied it, and the top one won. Dashes become dots the same way underscores do. When a flag’s name and its key genuinely differ, BindFlag is the way.

The bit worth stopping on: a flag you didn’t type contributes nothing

server-port has a declared default of 0. Run without it:

$ go run . --log-level warn
server.port      = 9090       server.port = 9090 (from config.yaml); also defined in default:embedded:defaults.yaml
log.level        = warn       log.level = warn (from flag:--log-level); also defined in default:embedded:defaults.yaml, config.yaml

server.port is 9090, not 0. Only flags actually set on the command line join the layer, because the store asks pflag which flags were visited rather than reading what every flag’s value happens to be. Skip that and every unset flag’s zero value sits at the top of the stack burying every layer underneath, which makes the whole flags layer worse than useless.

It’s also why declared flag defaults are the wrong home for configuration defaults. Put those in the defaults layer, as in step 2, where anything can override them.

6. Ask which layer won, and which lost

Explain is the readable form you’ve been printing all along. Two lower-level calls give you the same facts as data, which is what you want when you’re rendering them somewhere:

	if src, ok := store.View().Origin("server.port"); ok {
		fmt.Printf("winner: %s (writable: %v)\n", src, src.Writable)
	}
	for _, s := range store.View().Shadowed("server.port") {
		fmt.Printf("shadowed: %s\n", s)
	}

Origin names the layer whose value you actually get. Writable on that Source is the field the next step turns on.

A divergence I hit here, so you don’t lose ten minutes to it. The docs say Shadowed lists the sources that defined the key and lost. At v0.17.2 it returns the winner as well:

winner: env:CFGDEMO_SERVER_PORT (writable: false)
shadowed: default:embedded:defaults.yaml
shadowed: config.yaml
shadowed: env:CFGDEMO_SERVER_PORT

Clearest on a key only one source sets, where Shadowed names that source as having been shadowed by nothing at all. Explain gets it right on the same run, listing two sources under also defined in, not three. Two calls over one store, disagreeing about who lost. Raised as go/config#10; until it settles, trust Explain and treat Shadowed as every source that defines the key.

7. Write a value back, and watch routing skip what it can’t write

Here’s where the defaults layer being unwritable stops being trivia. server.timeout exists only in the compiled-in defaults, a layer with nowhere to persist to. Ask where a change would go before making it:

	plan, err := store.Plan(config.Set("server.timeout", 60))
	if err != nil {
		log.Fatal(err)
	}
	for _, op := range plan.Operations {
		fmt.Printf("plan: %s -> %s (effective: %v)\n", op.Change.Path, op.Target, op.Effective())
	}

plan: server.timeout -> config.yaml (effective: true)

Routing walked past the defaults because they can’t be written, and picked the highest layer that can. Apply it, and config.yaml becomes:

# Deployment settings for the demo service.
server:
  # The port the operators agreed on.
  port: 9090
  timeout: 60
log:
  level: debug

Both comments survived, port kept its place, and timeout landed in the section it belongs to. The file was edited, not regenerated from a parsed tree, which is the difference between a tool an operator will let near their config and one they’ll ban after the first time it eats their comments.

8. Write a value that a higher layer still shadows

Which brings us back to the call at the top. Write server.port while the environment is also setting it:

$ CFGDEMO_SERVER_PORT=7000 go run .
plan: server.port -> config.yaml (effective: false)
      shadowed by [env:CFGDEMO_SERVER_PORT]

The write isn’t refused. config.yaml really does get port: 5555, and I checked. But the plan told you, before you committed to it, that reading the key back will still give you 7000:

	if !op.Effective() {
		fmt.Printf("      shadowed by %s\n", op.ShadowedBy)
	}

That’s the whole afternoon I mentioned, handed to you as a boolean. The setting saved, the file is correct, the running process ignores it, and instead of three people staring at a file that is definitely right, something can say out loud which layer is winning and why.

What you built

Exactly one of the four can be written to, which is the only reason routing has a decision to make.

From here, Configure a service from Consul adds a remote layer that behaves precisely like the four above, and Precedence and merge is the reasoning underneath all of it, including what happens to lists and maps.

Four layers is not the hard part. Being able to answer why is.