Telemetry on PHP Boy Scout

The consent you can't ask for

Sat, 06 Jun 2026 00:00:00 +0000

There’s a comfortable story going round about telemetry, and it goes like this. There are two kinds. There’s the creepy kind, the usage data a vendor harvests to work out who you are and what you do, and that kind needs your permission. And there’s the innocent kind, the operational data a service emits so the people running it can keep it up, and that kind is just plumbing, nobody’s business, no permission required. Two neat boxes, and only one of them has a lock on it.

I don’t think the boxes are that neat. And I think a fair few of the people drawing them that way know it.

Because there’s no clean line where operational data stops being personal. A web service’s logs carry IP addresses. Its traces carry the path you walked through the system, the ids of the things you touched, sometimes the very fields you sent. Point at almost any of it and a GDPR lawyer will cheerfully tell you it can be personal data, and that the law doesn’t much care whether you filed it under “analytics” or “observability”. The word you picked to describe the data was never the thing that decided whether it was personal. The data decided that, and a lot of operational data is personal.

So if you can’t hide behind the box marked “just plumbing”, what do you actually do?

Where I’m coming from

I should say up front that I haven’t always been this relaxed about it. I spent a good few years in righteous fury at every tool that phoned home, every “we collect anonymous telemetry to improve the product” I never agreed to. Then I started building the tools, and I needed the data myself: the kind that tells you which features people actually use and which command falls over on first run, the kind that lets you make the next decision with something better than a hunch. And it softened me. Not into thinking it’s fine to take it without asking. Into understanding why everyone wants to.

What the fury left me with, the one thing I’ve never talked myself out of, is being pro-choice. Not pro-collection, not anti. Pro-choice. Any tool I ask another person to run will never quietly opt them into sending me a thing. It asks. On first run it makes its case, says what it wants and why, and lets them say no and mean it. I’ll try hard to win the yes, because the data is genuinely useful and a tool gets better when people share it. But I won’t presume it. The choice is theirs, and the prompt exists so they actually get to make it.

The trouble with a service

Which is a lovely principle right up until you build a web service. Because who, exactly, do you prompt? An API doesn’t have a first run. It has a thousand callers a second, none of them sat at a terminal waiting to tick a box. You can’t show a consent dialog to a webhook. The answer the industry reaches for is “consent is implied by use”, and… maybe. It’s a grey area, full stop. Implied consent is the same hand-wave that gave us the cookie banner, the thing we all click through without reading. I’m not going to stand here and call it clean.

But there’s a version of the principle that survives the grey, and it’s the one I built the framework around. Consent belongs to whoever can actually give it. For a command-line tool, that’s the person running it, so you ask them. For a web service, the person who can give it was never the end user at all, because you can’t reach them. It’s the engineer who deploys the thing. They know what their service collects, who its users are, which law they sit under, whether they owe anyone a privacy notice. They are the one party in the whole chain who can make the call with any of the facts in front of them. So that’s where the choice goes.

Which is why, in go-tool-base, the web-service telemetry is a switch. On or off, the engineer’s hand on it, collecting only what you need to keep the lights on by default. There’s no consent prompt, not because consent stopped mattering, but because there’s nobody in the loop I could ask. The accountability sits with the person who can hold it.

The part I’ll own

I’m pro-choice on telemetry, which is exactly why I built a way to switch it off and a way to force it on. Because for a web service the person holding the choice was never the end user, it’s the engineer who ships it, and “pro-choice” has to mean putting the switch in their hand, not pretending a popup would have meant anything.

That force-it-on part is the bit I’ll answer for. I built a way for a tool author to bypass the first-run prompt entirely and bake the consent in. There’s a real use case behind it, the enterprise tool deployed under a policy where collection is contractual rather than optional. But I also know I’ve handed someone a way to take the choice away, and I did it deliberately. Rightly or wrongly, I made the framework flexible enough to do the wrong thing, and the line I care about is now only as safe as the judgement of whoever picks it up.

That’s the uncomfortable place this lands, and I’ve come to think it’s the true one. A framework can put the choice in the right hands. It cannot make the right choice. I can build the prompt, build the switch, set the defaults to the modest thing, and after that I have to trust the engineer on the other side to use it justly and with some wisdom, because there is nothing further down the stack that makes them. When the blame gets shared out, and it’s always shared, a piece of it has my name on it, for every escape hatch I left in.

I’m at peace with that, mostly. Not because the grey went away, but because the alternative, pretending there’s a clean line and that “operational” means “not your problem”, is the real dodge. I’d rather say it plainly: this data can be personal, the consent is real even when there’s nobody to ask, and the most a tool can do is hand the decision to the person who can make it, and trust them with it.

Telemetry that asks, and telemetry that doesn't

Thu, 04 Jun 2026 00:00:00 +0000

go-tool-base has had a thing called telemetry for a long while now. It’s the opt-in kind: the product analytics that asks a user’s permission before it phones a single byte home, sits there as a no-op until they say yes, and can be wiped on request. The whole package is built around consent.

Then the web-service series went and needed telemetry too. Not that telemetry. The other one, the one the rest of the industry means when it says the word: traces, metrics and logs of a running service. And the awkward thing about those two is that they share a name, they want to share a package, and they pull in exactly opposite directions on the one question that matters most.

This is the story of how 0.7.x grew a second telemetry without breaking the first, and where the line between them ended up getting drawn.

Why bother putting it in the framework at all

The starting point is that I could have left observability out. A reader could wire up OpenTelemetry in their own service and go about their day. But the six parts of the web-service series spent a lot of effort making the transports first-class: a gRPC server, an HTTP server, a gateway, TLS across all of them, each one a Register call against the controller. Turning a CLI into a real long-running service and then shrugging “observability is your problem” would have left a hole exactly where it hurts.

Because a service you can’t see into is a liability the moment it leaves your laptop. The series ended with a macguffin service that was typed, fast and served over TLS, and was also a black box: when it got slow, you had nowhere to look. Metrics and traces are how you get the lights on, and they deserved the same first-class treatment as the things they observe.

The other half of the reason is that the framework already had a foot in this world. The analytics package’s preferred backend speaks OTLP, the OpenTelemetry wire protocol. So OpenTelemetry was already in the building. Doing observability any other way would have meant two standards where one would do.

The catch: two telemetries, opposite instincts

Here’s where it gets interesting, and it’s the part worth slowing down on.

The analytics telemetry is about a user. It collects usage data, hashed machine id, which command ran, exit code, and the entire design assumes you have to ask first. It is off by default. The collector you get when it’s disabled is a no-op, so nothing is recorded until the user opts in, and there’s a deletion path for when they change their mind. That’s not an add-on, that’s by design.

The observability telemetry is about a service. It emits operational data, how long a request took, which span was slow, how many errored, to a collector the operator runs. And there is no user in the loop to ask. The operator deploys the service, points it at their collector, and that act is itself the consent. Asking would be nonsensical: whose permission, for data about their own service, on their own infrastructure?

So you have two things called telemetry, wanting to live in one package, with the opposite default on consent. One is off until someone says yes; the other is on the moment it’s configured. Get that wiring wrong and you fail in one of two ugly ways. Gate the operational telemetry behind the user’s analytics opt-in, and a service’s tracing silently does nothing because nobody ticked a box meant for something else. Or loosen the analytics gate to make observability flow, and you start leaking usage data the user never agreed to share. Neither is acceptable, and “just use two packages” throws away everything the two genuinely have in common.

Quite a lot, as it turns out, and all of it below the consent line.

Both ship their data over OTLP to a collector. Both need to describe who is emitting, the service name and version, the resource in OpenTelemetry’s terms. Both parse an endpoint, attach headers, decide whether the connection is plaintext. None of that has the faintest thing to do with consent. It’s just the plumbing of getting bytes to a collector, and the analytics backend already had all of it, written inline.

So the shape of the solution fell out of the problem. Lift the shared plumbing into one place, let both telemetries stand on it, and keep the consent decision firmly out of that shared layer. The structure under pkg/telemetry ended up like this:

pkg/telemetry/
 telemetry.go the analytics Collector (consent-gated)
 backend_otel.go its OTLP backend
 posthog/ datadog/ vendor analytics backends
 otelcore/ shared: OTLP endpoint, resource, telemetry.* config
 tracing/ observability signal
 metrics/ observability signal
 logs/ observability signal
 observability.go Setup: builds the enabled signals (implied consent)

The new otelcore is the keystone. It holds the three things both sides need and nothing they don’t: ParseEndpoint for the OTLP URL, Resource for the service identity, and Resolve for reading the shared telemetry.* config (a base endpoint, plus per-signal overrides, in the same cascade as the TLS config). It imports no signal exporter and knows nothing about traces, metrics, logs or analytics. It is deliberately dumb plumbing.

The refactor: making the old telemetry stand on the new core

This next part is where the old telemetry and the new one become a single thing. The analytics OTLP backend was the first user of OTLP in the framework, and it had grown its own copy of all that plumbing: a function that parsed the endpoint URL, split out the host and path, worked out the insecure flag, and built the resource from a service name. Exactly the code the three new signals were about to need.

So rather than write it a second time and let the two drift, the analytics backend was refactored onto otelcore. Its exporter builder, buildOTelExporterOpts, now calls otelcore.ParseEndpoint, the same function tracing, metrics and logs call, and the resource comes from otelcore.Resource, the same one they use. One implementation of “talk OTLP to a collector”, four callers: the analytics backend and the three observability signals. Change how the framework forms an OTLP endpoint, and every signal moves together.

The reassuring part was that the analytics tests didn’t budge. The refactor moved code without changing behaviour, and the consent machinery, the opt-in, the no-op-when-disabled, the deletion path, never came near otelcore. Which is exactly the point.

Where the line is

Because the shared core is the easy half. The half that earns its keep is the bit that isn’t shared, and it’s a single, deliberate line.

The analytics collector keeps its gate. The constructor, NewCollector, still returns a no-op the moment telemetry is disabled, so a user who hasn’t opted in gets a collector that silently discards everything. Informed consent, untouched.

Observability gets a different door entirely. Setup builds whichever signals the operator has switched on, and it is gated only by telemetry.tracing.enabled and its siblings, which the operator sets. It never consults the analytics opt-in. Turning on tracing doesn’t turn on analytics; disabling analytics doesn’t silence tracing. The two enable flags live under the same telemetry.* config root, sit next to each other, and never read each other.

So that’s the whole architecture in a sentence: one package, one OTLP export core, two consent models that share everything except the answer to “do we need to ask”. The principle underneath, the one that decided every one of these calls, is that the kind of data sets the consent model. Usage data about a person needs informed consent. Operational data about a service runs on implied consent. The CLI and the web service are just where each kind tends to live.

Where this leaves the framework

0.7.x came out the other side with both telemetries: the one that asks first, exactly as it was, and a new one that doesn’t, because it has nobody to ask. They share an export core, a config root and a name, and they part company on the only thing they were ever going to disagree about.

I’ve been careful here to describe how the two consent models are kept apart, not to argue why they have to be. That argument, that “the kind of data decides the consent model” is a line worth holding rather than a convenient bit of engineering, is a piece of its own, and it’s the one I’m writing next.

Two telemetry events, one mangled line

Sun, 03 May 2026 00:00:00 +0000

A line in a log file that no parser would touch. Not a wrong value, not a missing field. Half of one telemetry event spliced into the middle of another, like two people typing into the same text box at once. Which, it turns out, is pretty much exactly what had happened.

A format with exactly one rule

rust-tool-base writes its telemetry to a file as JSONL: one JSON object per line, newline at the end, next object on the next line. It’s a lovely format to work with precisely because it has one rule, and the rule is simple. Every line is a complete object. Honour that and you can tail it, grep it, stream it into anything. Break it once and the whole file is suspect, because now a reader can’t trust that a line is a line.

So the one job the file sink has, beyond writing the right bytes, is to never let two events end up sharing a line.

“Appending is atomic, though, isn’t it?”

The mental model I started with, and I suspect I’m not alone, was this: open the file with O_APPEND, write the serialised event, and the operating system tacks it onto the end atomically. Two writers can’t tread on each other because each write goes to wherever the end currently is, no questions asked. I’d half-remembered O_APPEND as the thing that makes concurrent appending safe, full stop.

It’s half true, and the half that’s missing is the half that bit me.

O_APPEND does guarantee one thing: the seek-to-end and the write happen as a unit, so you never get the classic lost-update where two writers compute the same offset and clobber each other. Good. What it does not guarantee, on POSIX, is that a single write() of arbitrary size is atomic with respect to other writers. That atomicity has a ceiling, and the ceiling is PIPE_BUF: 4096 bytes on Linux. Under it, a write is all-or-nothing against other writes to the same file. Over it, the kernel is entirely within its rights to split your write into chunks, and another writer’s bytes can land in the gap between them.

The fat event that went over the edge

For a long time nothing went wrong, which is the most dangerous way for a bug like this to behave. A typical event, a command name, a duration, a status, an attribute or two, serialises to a few hundred bytes. Comfortably under four kilobytes, so comfortably inside the atomic window. Hundreds of them a day, never a problem.

Then an event came along with a lot of attributes on it, and its serialised form sailed past 4 KiB. Two of those emitted at roughly the same moment, both over the line, and O_APPEND did the only thing it had ever promised: it put each write at the end. It said nothing about not interleaving the bytes on the way, because past PIPE_BUF that was never on offer. One spliced line, one file a parser would now choke on.

The fix isn’t a bigger write, it’s a smaller gate

You can’t buy your way out of this with a bigger buffer, because there’s no buffer size that’s reliably atomic above PIPE_BUF. The fix is to stop relying on the kernel for mutual exclusion you can do yourself: serialise the events through a lock, so only one write is ever in flight at a time. The FileSink carries a mutex for exactly that, and the doc comment on it is the whole post in a paragraph, from crates/rtb-telemetry/src/sink.rs:

pub struct FileSink {
 path: PathBuf,
 // Serialises concurrent `emit` calls. Shared across `Clone`s of
 // the same `FileSink` so multiple handles to the same path also
 // serialise correctly.
 gate: Arc<tokio::sync::Mutex<()>>,
}

If you don’t write Rust day to day (the primer has the rest of the basics): tokio::sync::Mutex is an async-aware lock, .await is where a task waits its turn for that lock without blocking the whole thread, and the Arc wrapper is shared ownership. That Arc is the load-bearing bit: it means every clone of the FileSink points at the same gate, rather than each getting its own lock that guards nothing.

The detail I like is where the lock sits. The event is serialised to a string first, outside the critical section, because turning an event into JSON is the expensive part and there’s no reason to hold the gate while you do it. Only then does emit take the lock, and it holds it across the whole open-write-flush, so no other emit can interleave a single byte:

// Serialise the line outside the critical section.
let mut line = serde_json::to_string(&redacted)?;
line.push('\n');

let _guard = self.gate.lock().await;
// ...create parent dir, open with append(true)...
f.write_all(line.as_bytes()).await?;
f.flush().await?;

write_all makes sure the whole line goes out as one logical write from our side, and the gate makes sure ours is the only one happening. The 4 KiB cliff is still there in the kernel. We just never walk near it any more, because we’ve serialised the writers ourselves rather than hoping the OS would.

The bit even the lock can’t fix

There is however a genuine limit, and the comment is upfront about it. The mutex lives in the process. Two FileSinks in two different processes, both pointed at the same file, are back to relying on O_APPEND alone, and back under the 4 KiB ceiling. The lock can’t reach across a process boundary, so it doesn’t pretend to. The guidance there is the older, duller, correct one: give each process its own file and aggregate them somewhere else. Don’t have two processes fighting over one log file and expect the filesystem to referee.

What it comes down to

O_APPEND is a real guarantee, just a much smaller one than its name talks you into. It keeps your write at the end of the file, and it keeps concurrent writes from interleaving only while they stay under PIPE_BUF, which on Linux is 4096 bytes. A fat JSON event slides straight over that and takes your file’s one rule with it.

The fix was never exotic. Serialise the line, take a mutex, do the write under it, and the interleave can’t happen because there’s only ever one writer at a time. The POSIX manual had all of this written down long before I went and learned it the interesting way, which is, I’m told, how most people meet PIPE_BUF too.

Telemetry that asks first

Mon, 30 Mar 2026 00:00:00 +0000

Usage telemetry is genuinely useful. Knowing which commands people actually run, where the errors cluster, whether anyone ever touched the feature you spent a fortnight on… that’s the stuff that makes you a better maintainer. Wanting it is completely legitimate.

The trouble is that the usual way of getting it, on by default and quietly hoovering up everything, is a small betrayal of the people who installed your tool to get a job done. I wasn’t willing to build that, so go-tool-base’s telemetry starts from a different question.

The data you want, and the line you shouldn’t cross

If you maintain a tool, you want to know how it’s actually used. Which commands matter and which are dead weight. Where the error rate spikes. Whether anyone touched the feature you spent that fortnight on. That information makes you a better maintainer, and, to say it again, wanting it is completely legitimate.

The trouble is the standard way of getting it. Telemetry on by default. An opt-out buried three levels down in a settings file nobody reads. And once it’s running, it quietly collects far more than it ever admitted to: the arguments people passed, the paths they were working in, an IP address for good measure.

Every one of those is a small betrayal of someone who installed your tool to get a job done, not to become a data point. And the cost when users notice isn’t a slap on the wrist. It’s trust, and trust in a developer tool does not grow back quickly. A tool that surprises you once with what it was quietly collecting is a tool you uninstall and warn your colleagues about.

So go-tool-base’s telemetry started from a different question. Not “how do we collect the most data” but “how do we collect useful data without ever putting the user in a position they didn’t choose”.

Rule one: it is off until you say otherwise

The foundation is the simplest possible rule, and it’s absolute. Telemetry is never enabled by default. A freshly installed tool built on go-tool-base sends nothing. Not a heartbeat, not a ping, nothing at all.

It only starts collecting when the user makes an explicit, visible choice to let it. Three honest doors: they run telemetry enable, they say yes to a clear prompt during init, or they set TELEMETRY_ENABLED themselves. All three are deliberate acts. None of them is a pre-ticked box or a default they have to discover and then undo.

This is opt-in, and the distinction from a well-hidden opt-out is the entire point. Opt-out telemetry treats consent as something to be assumed and grudgingly reversed. Opt-in treats it as something that has to be given. Only one of those is actually consent.

Rule two: no personally identifiable information, full stop

Consent to “some telemetry” is not consent to “any telemetry”, so the second rule constrains what can ever be collected, even from a user who’s opted in.

No personally identifiable information. The framework does not record command arguments (they routinely contain paths, hostnames, the occasional secret someone’s pasted in). It does not record file contents. It does not record IP addresses.

It does need some notion of “distinct installations” for the numbers to mean anything, so it derives a machine ID from a handful of system signals and runs it through SHA-256. What leaves the machine is a hash. It tells you “this is the same install as last week” and tells you precisely nothing about whose install it is, and the hash can’t be walked backwards into the signals it came from.

The events themselves are deliberately thin. Which command ran, roughly how long it took, whether it errored. The shape of usage, not a transcript of it.

Rule three: the author picks the destination

Even with consent given and PII excluded, there’s a third question: where does the data actually go? go-tool-base doesn’t answer that for you, because it can’t. A corporate internal tool, an open-source CLI and an air-gapped utility have completely different right answers.

So the backend is the tool author’s choice. The framework ships several (a noop backend, stdout, a file, plain HTTP, and OpenTelemetry over OTLP) and supports custom ones. The noop backend matters more than it looks: it lets a tool wire up the whole telemetry surface, commands and all, while sending data precisely nowhere. A perfectly reasonable, fully supported configuration.

Pluggable backends also mean the data never has to touch any infrastructure I run. It goes where the tool’s author decides, on their terms. The framework provides the plumbing and stays well out of the destination.

And a way back out

One last thing, because it’s the part that makes the opt-in real rather than decorative. A user who opted in can opt straight back out, and the package includes a GDPR-aligned deletion path, so “stop, and remove what you have” is an actual supported request rather than a polite fiction.

Consent you can’t withdraw isn’t consent. It’s a one-way door with a friendly sign on it. The deletion path is what keeps the front door an actual door.

The bottom line

Telemetry is genuinely useful to a maintainer and genuinely dangerous to the trust of the people running the tool, and the usual implementation (on by default, opt-out buried, collecting everything) spends that trust recklessly. go-tool-base’s telemetry holds three lines: never enabled without an explicit user action, never collecting personally identifiable information even once enabled, and always sending data to a destination the tool’s author chose, up to and including nowhere. A real deletion path makes the opt-in something you can take back.

You can have your usage numbers. You just have to ask for them, the way you would for anything else that wasn’t yours to begin with.