I build my web services gRPC-first and bolt the REST on afterwards. Define the service once in protobuf, get your typed server and your generated clients for free, and then put a grpc-gateway in front so the callers who want plain JSON over HTTP can have it without you maintaining a second hand-written API. One source of truth, two front doors. I’m a big enough believer in this shape that go-tool-base ships the gateway as a first-class transport, so you get the REST surface by wiring an option, not by writing a translation layer yourself.

I’d just finished plumbing in proper observability: OpenTelemetry-native traces, metrics and logs, all going out over OTLP. And like any sensible person who’s just announced to himself “right, we have observability now”, I did the one thing that announcement demands. I built a tiny throwaway service to check the telemetry actually did what I’d just claimed it did.

It didn’t. Or rather, it did, but in two halves.

A request would come in over REST, hit the gateway, get translated into a gRPC call against my own server, and run. And in the trace viewer I’d get… a span for the gateway. And, entirely separately, an orphaned little trace for the gRPC handler, floating off on its own with no parent, as if it had happened in a different universe. The one request, the one logical operation, showing up as two unrelated things. The exact failure that makes a trace worthless: it tells you both halves happened, and nothing about them being the same story.

Here’s the bit that stung. The gateway translates HTTP into gRPC by dialling your gRPC server, in-process, and making the call. And that dial was not carrying the trace context. The incoming request had a perfectly good span going; the moment the gateway turned around and dialled gRPC, it started fresh, because nobody had told the outgoing call about the context coming in. So the server handler began a new trace instead of continuing the existing one.

The thing I’d built to see my system was the one place my system went blind.

The fix is small and slightly embarrassing in its smallness: propagate the incoming context through the gateway’s gRPC dial, so the outgoing call inherits the trace and the server span hangs off the gateway span where it belongs. One join, restored. The trace went from two strangers to one continuous line through the request, REST edge to gateway to handler, the way it should always have read.

The convenient layer is always where this sort of thing goes missing, and it goes missing precisely because it’s convenient. The gateway’s entire job is to hand you a REST surface so you never have to think about the join between HTTP and gRPC, but “you don’t have to think about it” and “nothing falls through it” turn out to be very different promises, and trace context is exactly the kind of thing that slips through a join you’ve stopped looking at.

The only reason I caught it at all wasn’t cleverness. I won’t trust telemetry I haven’t watched work with my own eyes, on a real request, end to end, and never will. Not in production, not off an alert. Observability you’ve wired up and never actually followed through is a guess wearing a dashboard, and that throwaway service that felt like five wasted minutes was the only thing keeping mine from lying to me.

If you want the longer version of how the observability itself is put together, that’s the subject of its own write-up. This is just the story of the day I found out the bit doing the translating was also the bit doing the forgetting.