<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
    <channel>
        <title>Go-Observability on PHP Boy Scout</title>
        <link>https://phpboyscout.uk/categories/go-observability/</link>
        <description>Recent content in Go-Observability on PHP Boy Scout</description>
        <generator>Hugo -- gohugo.io</generator>
        <language>en-gb</language>
        <copyright>Matt Cockayne</copyright>
        <lastBuildDate>Wed, 18 Nov 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://phpboyscout.uk/categories/go-observability/index.xml" rel="self" type="application/rss+xml" /><item>
            <title>Five requests, three queries</title>
            <link>https://phpboyscout.uk/five-requests-three-queries/</link>
            <pubDate>Wed, 18 Nov 2026 00:00:00 +0000</pubDate>
            <guid>https://phpboyscout.uk/five-requests-three-queries/</guid>
            <description>&lt;img src=&#34;https://phpboyscout.uk/five-requests-three-queries/cover-five-requests-three-queries_hu_1b80fde95d3ca465.jpg&#34; alt=&#34;Featured image of post Five requests, three queries&#34; /&gt;&lt;p&gt;A small service will happily answer &lt;code&gt;curl&lt;/code&gt; all day without telling you much about how it got there. You can see that it replied. What you can&amp;rsquo;t see is whether the cache in front of the database is doing anything at all, how long the query took, or how many of those requests failed without a word.&lt;/p&gt;&#xA;&lt;p&gt;Metrics are how you find out, and the catch is that a raw OpenTelemetry meter hands you a counter and leaves the names, the units and the labels up to you&amp;hellip; and then you get to make all of those decisions again in the next service, slightly differently.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;instrument&lt;/code&gt; package in &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;go/observability&lt;/a&gt; comes with those decisions already made, and this walkthrough is a gentle tour of it. We&amp;rsquo;ll build a small service from scratch, then add metrics to it a stage at a time, running it after each stage so you can see what each new piece reports before moving on to the next.&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-were-going-to-do&#34;&gt;What we&amp;rsquo;re going to do&#xA;&lt;/h2&gt;&lt;p&gt;The service is about as small as a real one gets. It answers &lt;code&gt;GET /orders/{id}&lt;/code&gt; with the name of the customer who placed that order. The orders live in a SQLite database, and the service keeps each customer name it has already looked up in a map (a very simple cache), so asking for one order twice only bothers the database once.&lt;/p&gt;&#xA;&lt;p&gt;Over the course of the tutorial we&amp;rsquo;ll:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Build the service with no metrics at all&lt;/strong&gt;, and check it works.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Teach it to stop cleanly&lt;/strong&gt; when you press Ctrl+C, because that&amp;rsquo;s when it will print its metrics.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Give it a meter and measure each request&lt;/strong&gt;: how many there were, how many failed and how long they took.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Watch the database&lt;/strong&gt;: how many queries reached it, and whether its connection pool is under pressure.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Watch the cache&lt;/strong&gt;: how often the map already held the name.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Send it five requests and read the whole story&lt;/strong&gt; it tells about them.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Each stage builds on the last, and the service compiles and runs at the end of each one, so if something goes wrong you&amp;rsquo;ll know which stage it went wrong in.&lt;/p&gt;&#xA;&lt;h3 id=&#34;a-few-words-before-we-start&#34;&gt;A few words before we start&#xA;&lt;/h3&gt;&lt;p&gt;If you&amp;rsquo;ve not worked with metrics before, here&amp;rsquo;s the vocabulary we&amp;rsquo;ll lean on. Feel free to skip this if it&amp;rsquo;s familiar.&lt;/p&gt;&#xA;&lt;p&gt;A &lt;strong&gt;metric&lt;/strong&gt; is a number your program keeps up to date as it runs, like a car&amp;rsquo;s mileage counter, rather than a line written for each event the way a log is. Once in a while the program reports the current numbers somewhere. That&amp;rsquo;s cheap, it doesn&amp;rsquo;t grow with traffic, and it&amp;rsquo;s pretty good at questions like &amp;ldquo;how many?&amp;rdquo; and &amp;ldquo;how long, usually?&amp;rdquo;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;OpenTelemetry&lt;/strong&gt; (often shortened to OTel) is the open standard most tools now use for metrics, traces and logs. It&amp;rsquo;s worth a search if the name is new to you. In OTel terms you&amp;rsquo;ll meet three things in this tutorial:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a &lt;strong&gt;provider&lt;/strong&gt;, which owns all the metrics and decides when and where they&amp;rsquo;re reported;&lt;/li&gt;&#xA;&lt;li&gt;a &lt;strong&gt;meter&lt;/strong&gt;, the handle you ask for from the provider and create your metrics from;&lt;/li&gt;&#xA;&lt;li&gt;an &lt;strong&gt;exporter&lt;/strong&gt;, which is where the provider sends them. Normally that&amp;rsquo;s a collector over the network. Here it&amp;rsquo;ll be your terminal.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Finally, two acronyms that give the &lt;code&gt;instrument&lt;/code&gt; package its shape. &lt;strong&gt;RED&lt;/strong&gt; (rate, errors, duration) is what you measure about &lt;em&gt;work&lt;/em&gt;: how often it happens, how often it fails and how long it takes. &lt;strong&gt;USE&lt;/strong&gt; (utilisation, saturation, errors) is what you measure about &lt;em&gt;resources&lt;/em&gt;, such as a connection pool: how busy it is, and whether anything is queueing for it. The module&amp;rsquo;s own &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/explanation/red-and-use/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;explanation page&lt;/a&gt; goes into both properly.&lt;/p&gt;&#xA;&lt;h3 id=&#34;what-youll-need&#34;&gt;What you&amp;rsquo;ll need&#xA;&lt;/h3&gt;&lt;ul&gt;&#xA;&lt;li&gt;Go at the version on the &lt;code&gt;go&lt;/code&gt; line of &lt;a class=&#34;link&#34; href=&#34;https://gitlab.com/phpboyscout/go/observability/-/blob/main/go.mod&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;go/observability&amp;rsquo;s &lt;code&gt;go.mod&lt;/code&gt;&lt;/a&gt;, or later. That moves as the module does, so check it rather than trusting a number here.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;curl&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Two terminals: one to run the service in, one to send it requests.&lt;/li&gt;&#xA;&lt;li&gt;About forty minutes, a bit more if you stop to read the output carefully (which is rather the point).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;You don&amp;rsquo;t need a collector, Docker, a dashboard or go-tool-base. It&amp;rsquo;s a plain &lt;code&gt;net/http&lt;/code&gt; service.&lt;/p&gt;&#xA;&lt;p&gt;This post is a slower, more explained version of the canonical tutorial in the go/observability docs, which is the copy kept up to date as the module moves: &lt;strong&gt;&lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/tutorials/instrument-a-small-service/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;https://observability.go.phpboyscout.uk/tutorials/instrument-a-small-service/&lt;/a&gt;&lt;/strong&gt;. Each stage below was run end to end against &lt;strong&gt;go/observability v0.6.0&lt;/strong&gt; and go/errors v0.4.0 on Go 1.27.3, and the output shown is what came back.&lt;/p&gt;&#xA;&lt;p&gt;If you followed &lt;a class=&#34;link&#34; href=&#34;https://phpboyscout.uk/building-a-web-service-with-go-tool-base-part-6/&#34; &gt;part 6 of the web service series&lt;/a&gt;, you&amp;rsquo;ve already seen traces, which follow one request through each hop it makes. Metrics come at it from the other side: they tell you about all the requests added up, which is what you want when the question is &amp;ldquo;is the cache earning its memory?&amp;rdquo; rather than &amp;ldquo;why was that one request slow?&amp;rdquo;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-1-create-the-project&#34;&gt;Step 1: Create the project&#xA;&lt;/h2&gt;&lt;p&gt;Make a directory for the service, start a Go module in it, and fetch the three libraries we&amp;rsquo;ll use:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;mkdir checkout &amp;amp;&amp;amp; cd checkout&#xA;go mod init example.com/checkout&#xA;go get gitlab.com/phpboyscout/go/observability&#xA;go get gitlab.com/phpboyscout/go/errors&#xA;go get modernc.org/sqlite&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;go/observability&lt;/code&gt; holds the &lt;code&gt;instrument&lt;/code&gt; package and the plumbing to set up a provider.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;go/errors&lt;/code&gt; lets us give an error a &lt;em&gt;kind&lt;/em&gt;, a short fixed name for a type of failure. That turns out to matter for metrics, and we&amp;rsquo;ll see why in Step 4.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;modernc.org/sqlite&lt;/code&gt; is SQLite written in pure Go, so you don&amp;rsquo;t need a C compiler. It&amp;rsquo;s a big package, and the first build can take a minute or so.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;step-2-build-a-service-that-works-and-says-nothing&#34;&gt;Step 2: Build a service that works and says nothing&#xA;&lt;/h2&gt;&lt;p&gt;Before measuring anything, we want a service that works, so that when the numbers start appearing we know what they &lt;em&gt;should&lt;/em&gt; say. Create &lt;code&gt;main.go&lt;/code&gt; with this in it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;package main&#xA;&#xA;import (&#xA;&#x9;&amp;#34;context&amp;#34;&#xA;&#x9;&amp;#34;database/sql&amp;#34;&#xA;&#x9;&amp;#34;fmt&amp;#34;&#xA;&#x9;&amp;#34;log&amp;#34;&#xA;&#x9;&amp;#34;net/http&amp;#34;&#xA;&#x9;&amp;#34;strconv&amp;#34;&#xA;&#x9;&amp;#34;sync&amp;#34;&#xA;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/errors&amp;#34;&#xA;&#x9;_ &amp;#34;modernc.org/sqlite&amp;#34;&#xA;)&#xA;&#xA;// ErrNotFound has a kind, so a missing order is a queryable failure of its own.&#xA;var ErrNotFound = errors.NewSentinel(&amp;#34;order_not_found&amp;#34;, &amp;#34;no such order&amp;#34;)&#xA;&#xA;type service struct {&#xA;&#x9;conn *sql.DB&#xA;&#xA;&#x9;mu        sync.Mutex&#xA;&#x9;customers map[int]string&#xA;}&#xA;&#xA;func (s *service) customer(ctx context.Context, id int) (string, error) {&#xA;&#x9;s.mu.Lock()&#xA;&#x9;name, ok := s.customers[id]&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;if ok {&#xA;&#x9;&#x9;return name, nil&#xA;&#x9;}&#xA;&#xA;&#x9;err := s.conn.QueryRowContext(ctx, &amp;#34;SELECT customer FROM orders WHERE id = ?&amp;#34;, id).Scan(&amp;amp;name)&#xA;&#x9;if errors.Is(err, sql.ErrNoRows) {&#xA;&#x9;&#x9;return &amp;#34;&amp;#34;, ErrNotFound&#xA;&#x9;}&#xA;&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;return &amp;#34;&amp;#34;, err&#xA;&#x9;}&#xA;&#xA;&#x9;s.mu.Lock()&#xA;&#x9;s.customers[id] = name&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;return name, nil&#xA;}&#xA;&#xA;func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {&#xA;&#x9;id, err := strconv.Atoi(r.PathValue(&amp;#34;id&amp;#34;))&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;http.Error(w, &amp;#34;bad order ID&amp;#34;, http.StatusBadRequest)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;customer, err := s.customer(r.Context(), id)&#xA;&#x9;if errors.Is(err, ErrNotFound) {&#xA;&#x9;&#x9;http.Error(w, err.Error(), http.StatusNotFound)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;http.Error(w, &amp;#34;internal error&amp;#34;, http.StatusInternalServerError)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;fmt.Fprintln(w, customer)&#xA;}&#xA;&#xA;func main() {&#xA;&#x9;conn, err := sql.Open(&amp;#34;sqlite&amp;#34;, &amp;#34;file:checkout?mode=memory&amp;amp;cache=shared&amp;#34;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer conn.Close()&#xA;&#xA;&#x9;_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);&#xA;&#x9;&#x9;INSERT INTO orders VALUES (1, &amp;#39;Ada&amp;#39;), (2, &amp;#39;Grace&amp;#39;);`)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;s := &amp;amp;service{conn: conn, customers: map[int]string{}}&#xA;&#xA;&#x9;mux := http.NewServeMux()&#xA;&#x9;mux.HandleFunc(&amp;#34;GET /orders/{id}&amp;#34;, s.getOrder)&#xA;&#xA;&#x9;log.Println(&amp;#34;listening on :8080&amp;#34;)&#xA;&#x9;log.Fatal(http.ListenAndServe(&amp;#34;:8080&amp;#34;, mux))&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;That&amp;rsquo;s a fair bit of code in one go, so let&amp;rsquo;s walk through it from the top.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;ErrNotFound&lt;/code&gt;&lt;/strong&gt; is the error we return when an order doesn&amp;rsquo;t exist. &lt;code&gt;errors.NewSentinel&lt;/code&gt; gives it the kind &lt;code&gt;order_not_found&lt;/code&gt; as well as a message. For now it&amp;rsquo;s just an error; once metrics arrive, that kind becomes the label that tells &amp;ldquo;not found&amp;rdquo; apart from any other failure.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;service&lt;/code&gt;&lt;/strong&gt; holds the database connection and the cache. The cache is a plain map from order ID to customer name. Two requests can arrive at once, so the map sits behind a mutex (&lt;code&gt;mu&lt;/code&gt;) to stop them treading on each other.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;customer&lt;/code&gt;&lt;/strong&gt; is where the work happens. It checks the map first and returns straight away if the name is there. If it isn&amp;rsquo;t, it asks the database, turns SQL&amp;rsquo;s &amp;ldquo;no rows&amp;rdquo; error into our &lt;code&gt;ErrNotFound&lt;/code&gt;, and stores the name in the map for next time.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;getOrder&lt;/code&gt;&lt;/strong&gt; is the HTTP handler. It reads the ID from the URL, calls &lt;code&gt;customer&lt;/code&gt;, and turns the result into a response: the name, a 404 for an unknown order, or a 500 for anything else.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;main&lt;/code&gt;&lt;/strong&gt; opens the database, creates the &lt;code&gt;orders&lt;/code&gt; table with two orders in it, registers the handler and starts listening on port 8080.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The database is held in memory, so it vanishes when the program stops, which is fine for a tutorial. The &lt;code&gt;cache=shared&lt;/code&gt; part of the connection string makes all the connections in Go&amp;rsquo;s pool share one in-memory database, rather than each getting its own empty one.&lt;/p&gt;&#xA;&lt;p&gt;Run it, and leave it running:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;go run .&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;2026/11/13 12:25:48 listening on :8080&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;In your second terminal, ask for an order that exists and one that doesn&amp;rsquo;t:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;curl localhost:8080/orders/1&#xA;curl localhost:8080/orders/99&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;Ada&#xA;no such order&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;If you see &lt;code&gt;address already in use&lt;/code&gt;, something else already has port 8080 (it did on my machine). Change the port in &lt;code&gt;main.go&lt;/code&gt;, and in the &lt;code&gt;curl&lt;/code&gt; commands from here on.&lt;/p&gt;&#xA;&lt;p&gt;Stop the service with Ctrl+C. It works, and it tells us nothing about how.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-3-teach-the-service-to-stop-cleanly&#34;&gt;Step 3: Teach the service to stop cleanly&#xA;&lt;/h2&gt;&lt;p&gt;This step doesn&amp;rsquo;t add any metrics yet, but the metrics depend on it, so it&amp;rsquo;s worth doing on its own.&lt;/p&gt;&#xA;&lt;p&gt;The provider we&amp;rsquo;ll add in the next step reports its numbers every 60 seconds, and once more when it&amp;rsquo;s shut down. For a tutorial, that last report is what we want to read: send some requests, press Ctrl+C, and see the totals.&lt;/p&gt;&#xA;&lt;p&gt;Right now, though, Ctrl+C kills the program hard. Go&amp;rsquo;s default response to that signal is to exit immediately, so none of the clean-up gets a chance to run. A request that&amp;rsquo;s halfway through is cut off mid-reply. The &lt;code&gt;defer conn.Close()&lt;/code&gt; near the top of &lt;code&gt;main&lt;/code&gt; is skipped, and the database connection is simply abandoned. With an in-memory database in a tutorial that costs us very little, but a real service treated like this loses work every time it&amp;rsquo;s restarted, and in the next step it would also lose the provider&amp;rsquo;s last report before it&amp;rsquo;s ever written.&lt;/p&gt;&#xA;&lt;p&gt;The root of it is that &lt;code&gt;http.ListenAndServe&lt;/code&gt; only does half of a server&amp;rsquo;s job. It starts listening and serving requests, and then it blocks for as long as the program lives. There&amp;rsquo;s no way to ask it to stop, so the only way out is for the process to die. And even if it did return, &lt;code&gt;log.Fatal&lt;/code&gt; exits on the spot, which skips the deferred calls too.&lt;/p&gt;&#xA;&lt;p&gt;So in this step we give the server its other half. &lt;code&gt;main&lt;/code&gt; will notice Ctrl+C, ask the HTTP server to stop politely, wait for it to finish, and then carry on to the end of the function, so that everything deferred gets closed in order.&lt;/p&gt;&#xA;&lt;p&gt;Add &lt;code&gt;&amp;quot;os&amp;quot;&lt;/code&gt; and &lt;code&gt;&amp;quot;os/signal&amp;quot;&lt;/code&gt; to the imports, so the block reads:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;import (&#xA;&#x9;&amp;#34;context&amp;#34;&#xA;&#x9;&amp;#34;database/sql&amp;#34;&#xA;&#x9;&amp;#34;fmt&amp;#34;&#xA;&#x9;&amp;#34;log&amp;#34;&#xA;&#x9;&amp;#34;net/http&amp;#34;&#xA;&#x9;&amp;#34;os&amp;#34;&#xA;&#x9;&amp;#34;os/signal&amp;#34;&#xA;&#x9;&amp;#34;strconv&amp;#34;&#xA;&#x9;&amp;#34;sync&amp;#34;&#xA;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/errors&amp;#34;&#xA;&#x9;_ &amp;#34;modernc.org/sqlite&amp;#34;&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Then replace &lt;code&gt;main&lt;/code&gt; with this version:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;func main() {&#xA;&#x9;ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)&#xA;&#x9;defer stop()&#xA;&#xA;&#x9;conn, err := sql.Open(&amp;#34;sqlite&amp;#34;, &amp;#34;file:checkout?mode=memory&amp;amp;cache=shared&amp;#34;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer conn.Close()&#xA;&#xA;&#x9;_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);&#xA;&#x9;&#x9;INSERT INTO orders VALUES (1, &amp;#39;Ada&amp;#39;), (2, &amp;#39;Grace&amp;#39;);`)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;s := &amp;amp;service{conn: conn, customers: map[int]string{}}&#xA;&#xA;&#x9;mux := http.NewServeMux()&#xA;&#x9;mux.HandleFunc(&amp;#34;GET /orders/{id}&amp;#34;, s.getOrder)&#xA;&#xA;&#x9;server := &amp;amp;http.Server{Addr: &amp;#34;:8080&amp;#34;, Handler: mux}&#xA;&#xA;&#x9;go func() {&#xA;&#x9;&#x9;if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {&#xA;&#x9;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;&#x9;}&#xA;&#x9;}()&#xA;&#xA;&#x9;log.Println(&amp;#34;listening on :8080&amp;#34;)&#xA;&#x9;&amp;lt;-ctx.Done()&#xA;&#xA;&#x9;_ = server.Shutdown(context.Background())&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Three things changed:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;signal.NotifyContext&lt;/code&gt; gives us a context that&amp;rsquo;s cancelled when you press Ctrl+C, instead of the program being killed outright.&lt;/li&gt;&#xA;&lt;li&gt;The server now runs in its own goroutine, so &lt;code&gt;main&lt;/code&gt; is free to wait. &lt;code&gt;&amp;lt;-ctx.Done()&lt;/code&gt; is that wait: it blocks until Ctrl+C.&lt;/li&gt;&#xA;&lt;li&gt;When it comes, &lt;code&gt;server.Shutdown&lt;/code&gt; stops accepting new requests and lets any in-flight ones finish. &lt;code&gt;ListenAndServe&lt;/code&gt; then returns &lt;code&gt;http.ErrServerClosed&lt;/code&gt;, which is the normal &amp;ldquo;I was asked to stop&amp;rdquo; answer, so the goroutine only treats any &lt;em&gt;other&lt;/em&gt; error as fatal.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Run it again, send a request if you like, and press Ctrl+C:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;go run .&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;2026/11/13 12:26:10 listening on :8080&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This time the program returns you to the prompt without any error. It looks no different, but &lt;code&gt;main&lt;/code&gt; now reaches its last line, and that&amp;rsquo;s where the next step puts the metrics report.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-4-give-it-a-meter-and-measure-each-request&#34;&gt;Step 4: Give it a meter and measure each request&#xA;&lt;/h2&gt;&lt;p&gt;Now the first real metrics. We&amp;rsquo;ll set up a provider, get a meter from it, and use the &lt;code&gt;instrument&lt;/code&gt; package&amp;rsquo;s &lt;code&gt;Operations&lt;/code&gt; helper to measure the request.&lt;/p&gt;&#xA;&lt;p&gt;An &lt;em&gt;operation&lt;/em&gt; is just a piece of work you care about. Here it&amp;rsquo;s &amp;ldquo;serve an order&amp;rdquo;. &lt;code&gt;Operations&lt;/code&gt; creates two metrics for it:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;operation.duration&lt;/code&gt;&lt;/strong&gt;, a &lt;em&gt;histogram&lt;/em&gt; of how long each operation took. A histogram doesn&amp;rsquo;t keep each individual timing. It keeps a count, a total, the fastest and slowest, and how many fell into each of a set of time &lt;em&gt;buckets&lt;/em&gt;. The count gives you the rate, and counting only the samples labelled with an error gives you the errors, so this one metric covers all three letters of RED.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;operation.active&lt;/code&gt;&lt;/strong&gt;, how many operations are running at this moment.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;This step touches the file in four places, top to bottom. First the imports: add &lt;code&gt;log/slog&lt;/code&gt; and the three go/observability packages.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;import (&#xA;&#x9;&amp;#34;context&amp;#34;&#xA;&#x9;&amp;#34;database/sql&amp;#34;&#xA;&#x9;&amp;#34;fmt&amp;#34;&#xA;&#x9;&amp;#34;log&amp;#34;&#xA;&#x9;&amp;#34;log/slog&amp;#34;&#xA;&#x9;&amp;#34;net/http&amp;#34;&#xA;&#x9;&amp;#34;os&amp;#34;&#xA;&#x9;&amp;#34;os/signal&amp;#34;&#xA;&#x9;&amp;#34;strconv&amp;#34;&#xA;&#x9;&amp;#34;sync&amp;#34;&#xA;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/errors&amp;#34;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/observability/instrument&amp;#34;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/observability/metrics&amp;#34;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/observability/otelcore&amp;#34;&#xA;&#x9;_ &amp;#34;modernc.org/sqlite&amp;#34;&#xA;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Next, the &lt;code&gt;service&lt;/code&gt; struct gets a field to hold the operations helper:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;type service struct {&#xA;&#x9;conn *sql.DB&#xA;&#x9;ops  *instrument.Operations&#xA;&#xA;&#x9;mu        sync.Mutex&#xA;&#x9;customers map[int]string&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Then &lt;code&gt;getOrder&lt;/code&gt; wraps its call to &lt;code&gt;customer&lt;/code&gt; in &lt;code&gt;instrument.ObserveResult&lt;/code&gt;. Here&amp;rsquo;s the whole handler:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {&#xA;&#x9;id, err := strconv.Atoi(r.PathValue(&amp;#34;id&amp;#34;))&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;http.Error(w, &amp;#34;bad order ID&amp;#34;, http.StatusBadRequest)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;customer, err := instrument.ObserveResult(r.Context(), s.ops, &amp;#34;get_order&amp;#34;, func() (string, error) {&#xA;&#x9;&#x9;return s.customer(r.Context(), id)&#xA;&#x9;})&#xA;&#x9;if errors.Is(err, ErrNotFound) {&#xA;&#x9;&#x9;http.Error(w, err.Error(), http.StatusNotFound)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;http.Error(w, &amp;#34;internal error&amp;#34;, http.StatusInternalServerError)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;fmt.Fprintln(w, customer)&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;ObserveResult&lt;/code&gt; runs the function you give it, times it, and notes what kind of error came back (if any). Then it hands you the value and the error just as your function returned them, so the rest of the handler carries on unchanged. (It&amp;rsquo;s called &lt;code&gt;ObserveResult&lt;/code&gt; because it passes a value back. For work that only returns an error there&amp;rsquo;s a plain &lt;code&gt;Observe&lt;/code&gt;.)&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;&amp;quot;get_order&amp;quot;&lt;/code&gt; is the operation&amp;rsquo;s name, and it becomes a &lt;em&gt;label&lt;/em&gt; on each measurement. Labels are how you slice a metric later (&amp;ldquo;show me get_order, failures only&amp;rdquo;), but each distinct label value creates a separate series that whatever receives your metrics has to store. So names like this come from a small fixed set that you write in the code. Don&amp;rsquo;t build one from the order ID or anything else a user controls, or a busy day can turn into one series per order.&lt;/p&gt;&#xA;&lt;p&gt;Finally, &lt;code&gt;main&lt;/code&gt;. It now builds the provider at the top and shuts it down at the bottom:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;func main() {&#xA;&#x9;ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)&#xA;&#x9;defer stop()&#xA;&#xA;&#x9;logger := slog.New(slog.NewTextHandler(os.Stdout, nil))&#xA;&#xA;&#x9;provider, err := metrics.NewProvider(ctx,&#xA;&#x9;&#x9;otelcore.Resource(&amp;#34;checkout&amp;#34;, &amp;#34;0.1.0&amp;#34;),&#xA;&#x9;&#x9;otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},&#xA;&#x9;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;meter := provider.Meter(&amp;#34;checkout&amp;#34;)&#xA;&#xA;&#x9;conn, err := sql.Open(&amp;#34;sqlite&amp;#34;, &amp;#34;file:checkout?mode=memory&amp;amp;cache=shared&amp;#34;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer conn.Close()&#xA;&#xA;&#x9;_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);&#xA;&#x9;&#x9;INSERT INTO orders VALUES (1, &amp;#39;Ada&amp;#39;), (2, &amp;#39;Grace&amp;#39;);`)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;ops, err := instrument.NewOperations(meter)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;s := &amp;amp;service{conn: conn, ops: ops, customers: map[int]string{}}&#xA;&#xA;&#x9;mux := http.NewServeMux()&#xA;&#x9;mux.HandleFunc(&amp;#34;GET /orders/{id}&amp;#34;, s.getOrder)&#xA;&#xA;&#x9;server := &amp;amp;http.Server{Addr: &amp;#34;:8080&amp;#34;, Handler: mux}&#xA;&#xA;&#x9;go func() {&#xA;&#x9;&#x9;if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {&#xA;&#x9;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;&#x9;}&#xA;&#x9;}()&#xA;&#xA;&#x9;log.Println(&amp;#34;listening on :8080&amp;#34;)&#xA;&#x9;&amp;lt;-ctx.Done()&#xA;&#xA;&#x9;_ = server.Shutdown(context.Background())&#xA;&#xA;&#x9;if err := provider.Shutdown(context.Background()); err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Going through the new parts in order:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;The logger.&lt;/strong&gt; A standard &lt;code&gt;slog&lt;/code&gt; logger writing plain text to the terminal.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;The provider.&lt;/strong&gt; &lt;code&gt;metrics.NewProvider&lt;/code&gt; builds it. &lt;code&gt;otelcore.Resource(&amp;quot;checkout&amp;quot;, &amp;quot;0.1.0&amp;quot;)&lt;/code&gt; names the service and its version, so whatever receives the metrics knows who sent them. &lt;code&gt;otelcore.Settings&lt;/code&gt; switches metrics on and picks the exporter, and &lt;code&gt;otelcore.Slog(logger)&lt;/code&gt; is the exporter that writes each metric as a log line instead of sending it over the network.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;The meter.&lt;/strong&gt; &lt;code&gt;provider.Meter(&amp;quot;checkout&amp;quot;)&lt;/code&gt; is what the &lt;code&gt;instrument&lt;/code&gt; helpers create their metrics on.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;The operations helper.&lt;/strong&gt; &lt;code&gt;instrument.NewOperations(meter)&lt;/code&gt; creates the two metrics, and goes into the &lt;code&gt;service&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;provider.Shutdown&lt;/code&gt;.&lt;/strong&gt; At the very end, after the HTTP server has stopped, this makes the provider send its final report. That&amp;rsquo;s the export we&amp;rsquo;ll be reading.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Run it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;go run .&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;This time, in the second terminal, send it five requests: order 1 three times, order 2 once, and order 99, which doesn&amp;rsquo;t exist.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Go back to the first terminal and press Ctrl+C. The provider&amp;rsquo;s last report appears, one line per metric. Each line starts with &lt;code&gt;time=&lt;/code&gt; and &lt;code&gt;level=INFO&lt;/code&gt;, which I&amp;rsquo;ve left off here so the rest fits:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;msg=metric name=operation.duration unit=s kind=histogram count=1 sum=6.565e-05 min=6.565e-05 max=6.565e-05 buckets=&amp;#34;0.005=1&amp;#34; attrs.error.type=order_not_found attrs.operation.name=get_order&#xA;msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.00021654999999999998 min=3.1e-07 max=0.00012721 buckets=&amp;#34;0.005=4&amp;#34; attrs.operation.name=get_order&#xA;msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Here&amp;rsquo;s how to read it.&lt;/p&gt;&#xA;&lt;p&gt;The two &lt;code&gt;operation.duration&lt;/code&gt; lines are one metric with two different labels. The line with &lt;code&gt;attrs.error.type=order_not_found&lt;/code&gt; has &lt;code&gt;count=1&lt;/code&gt;: that&amp;rsquo;s the request for order 99. The line with no error label has &lt;code&gt;count=4&lt;/code&gt;: the other four. One plus four is all five requests, so you have the rate and the errors straight away.&lt;/p&gt;&#xA;&lt;p&gt;The duration lives in the other fields. &lt;code&gt;sum&lt;/code&gt; is the total time in seconds, &lt;code&gt;min&lt;/code&gt; and &lt;code&gt;max&lt;/code&gt; are the fastest and slowest, and &lt;code&gt;buckets=&amp;quot;0.005=4&amp;quot;&lt;/code&gt; says all four successful requests finished within 5 milliseconds. Your timings will differ (and the odd long string of 9s in a &lt;code&gt;sum&lt;/code&gt; is ordinary floating-point arithmetic), but the counts should match.&lt;/p&gt;&#xA;&lt;p&gt;Notice the error label says &lt;code&gt;order_not_found&lt;/code&gt;, the kind we gave &lt;code&gt;ErrNotFound&lt;/code&gt; back in Step 2. That&amp;rsquo;s why it was worth giving the error a kind. An error without one is labelled &lt;code&gt;_OTHER&lt;/code&gt;, which is accurate and not much help when you&amp;rsquo;re trying to tell one failure from another. You can&amp;rsquo;t label with the error&amp;rsquo;s message instead, because messages often contain IDs and other per-request values, and each one would become a new series.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;operation.active&lt;/code&gt; is &lt;code&gt;0&lt;/code&gt;, because no request was running at the moment the provider reported. Under real load, that&amp;rsquo;s the number that tells you how much is in flight.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-5-watch-the-database&#34;&gt;Step 5: Watch the database&#xA;&lt;/h2&gt;&lt;p&gt;We now know how long requests take, but not where the time goes. The next question is what the database is doing. The &lt;code&gt;instrument&lt;/code&gt; package&amp;rsquo;s &lt;code&gt;DB&lt;/code&gt; helper measures two things:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Each query&lt;/strong&gt; (RED again): how many, how many failed, how long they took.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;The connection pool&lt;/strong&gt; (USE): how many connections are open, how many are busy, and whether anything has had to wait for one.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The imports don&amp;rsquo;t change this time, since &lt;code&gt;instrument&lt;/code&gt; is already there. The &lt;code&gt;service&lt;/code&gt; struct gets a &lt;code&gt;db&lt;/code&gt; field:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;type service struct {&#xA;&#x9;conn *sql.DB&#xA;&#x9;ops  *instrument.Operations&#xA;&#x9;db   *instrument.DB&#xA;&#xA;&#x9;mu        sync.Mutex&#xA;&#x9;customers map[int]string&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;customer&lt;/code&gt; wraps its query in &lt;code&gt;s.db.Query&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;func (s *service) customer(ctx context.Context, id int) (string, error) {&#xA;&#x9;s.mu.Lock()&#xA;&#x9;name, ok := s.customers[id]&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;if ok {&#xA;&#x9;&#x9;return name, nil&#xA;&#x9;}&#xA;&#xA;&#x9;err := s.db.Query(ctx, &amp;#34;get_order&amp;#34;, func() error {&#xA;&#x9;&#x9;err := s.conn.QueryRowContext(ctx, &amp;#34;SELECT customer FROM orders WHERE id = ?&amp;#34;, id).Scan(&amp;amp;name)&#xA;&#x9;&#x9;if errors.Is(err, sql.ErrNoRows) {&#xA;&#x9;&#x9;&#x9;return ErrNotFound&#xA;&#x9;&#x9;}&#xA;&#xA;&#x9;&#x9;return err&#xA;&#x9;})&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;return &amp;#34;&amp;#34;, err&#xA;&#x9;}&#xA;&#xA;&#x9;s.mu.Lock()&#xA;&#x9;s.customers[id] = name&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;return name, nil&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It works like &lt;code&gt;ObserveResult&lt;/code&gt;: give it a name and a function, and it times the function and records its error. The query name becomes a label as well, so the fixed-names rule from Step 4 applies: &lt;code&gt;&amp;quot;get_order&amp;quot;&lt;/code&gt; is fine, the SQL text with an ID in it isn&amp;rsquo;t.&lt;/p&gt;&#xA;&lt;p&gt;Look closely and you&amp;rsquo;ll see the &lt;code&gt;sql.ErrNoRows&lt;/code&gt; check has moved &lt;em&gt;inside&lt;/em&gt; the function. That&amp;rsquo;s so the helper sees &lt;code&gt;ErrNotFound&lt;/code&gt;, with its &lt;code&gt;order_not_found&lt;/code&gt; kind, instead of SQL&amp;rsquo;s own error, which has no kind and would be recorded as &lt;code&gt;_OTHER&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;And &lt;code&gt;main&lt;/code&gt; gains the database helper, between the operations helper and the &lt;code&gt;service&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;func main() {&#xA;&#x9;ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)&#xA;&#x9;defer stop()&#xA;&#xA;&#x9;logger := slog.New(slog.NewTextHandler(os.Stdout, nil))&#xA;&#xA;&#x9;provider, err := metrics.NewProvider(ctx,&#xA;&#x9;&#x9;otelcore.Resource(&amp;#34;checkout&amp;#34;, &amp;#34;0.1.0&amp;#34;),&#xA;&#x9;&#x9;otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},&#xA;&#x9;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;meter := provider.Meter(&amp;#34;checkout&amp;#34;)&#xA;&#xA;&#x9;conn, err := sql.Open(&amp;#34;sqlite&amp;#34;, &amp;#34;file:checkout?mode=memory&amp;amp;cache=shared&amp;#34;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer conn.Close()&#xA;&#xA;&#x9;_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);&#xA;&#x9;&#x9;INSERT INTO orders VALUES (1, &amp;#39;Ada&amp;#39;), (2, &amp;#39;Grace&amp;#39;);`)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;ops, err := instrument.NewOperations(meter)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;db, err := instrument.NewDB(meter, &amp;#34;orders&amp;#34;, instrument.WithDBSystem(&amp;#34;sqlite&amp;#34;))&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer db.Close()&#xA;&#xA;&#x9;err = db.RegisterPool(func() instrument.PoolStats {&#xA;&#x9;&#x9;stats := conn.Stats()&#xA;&#xA;&#x9;&#x9;return instrument.PoolStats{&#xA;&#x9;&#x9;&#x9;MaxOpen:     stats.MaxOpenConnections,&#xA;&#x9;&#x9;&#x9;Open:        stats.OpenConnections,&#xA;&#x9;&#x9;&#x9;InUse:       stats.InUse,&#xA;&#x9;&#x9;&#x9;Idle:        stats.Idle,&#xA;&#x9;&#x9;&#x9;WaitCount:   stats.WaitCount,&#xA;&#x9;&#x9;&#x9;WaitSeconds: stats.WaitDuration.Seconds(),&#xA;&#x9;&#x9;}&#xA;&#x9;})&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;s := &amp;amp;service{conn: conn, ops: ops, db: db, customers: map[int]string{}}&#xA;&#xA;&#x9;mux := http.NewServeMux()&#xA;&#x9;mux.HandleFunc(&amp;#34;GET /orders/{id}&amp;#34;, s.getOrder)&#xA;&#xA;&#x9;server := &amp;amp;http.Server{Addr: &amp;#34;:8080&amp;#34;, Handler: mux}&#xA;&#xA;&#x9;go func() {&#xA;&#x9;&#x9;if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {&#xA;&#x9;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;&#x9;}&#xA;&#x9;}()&#xA;&#xA;&#x9;log.Println(&amp;#34;listening on :8080&amp;#34;)&#xA;&#x9;&amp;lt;-ctx.Done()&#xA;&#xA;&#x9;_ = server.Shutdown(context.Background())&#xA;&#xA;&#x9;if err := provider.Shutdown(context.Background()); err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;instrument.NewDB(meter, &amp;quot;orders&amp;quot;, instrument.WithDBSystem(&amp;quot;sqlite&amp;quot;))&lt;/code&gt; creates the query metrics for a database we&amp;rsquo;re calling &lt;code&gt;orders&lt;/code&gt;, and records that it&amp;rsquo;s SQLite.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;db.RegisterPool&lt;/code&gt; is how the helper finds out about the pool. The function you pass it is called each time the provider reports, and it copies Go&amp;rsquo;s own pool statistics (&lt;code&gt;conn.Stats()&lt;/code&gt;, from &lt;code&gt;database/sql&lt;/code&gt;) into the shape the helper expects. Keep it cheap, because it runs on every report.&lt;/p&gt;&#xA;&lt;p&gt;You might have wondered why &lt;code&gt;provider.Shutdown&lt;/code&gt; sits at the end of &lt;code&gt;main&lt;/code&gt; instead of being deferred. &lt;code&gt;defer db.Close()&lt;/code&gt; stops the pool reporting, and deferred calls run when &lt;code&gt;main&lt;/code&gt; returns. Had we written &lt;code&gt;defer provider.Shutdown(...)&lt;/code&gt; up at the top, Go would run it &lt;em&gt;last&lt;/em&gt; (deferred calls run in reverse order), after the pool had already stopped reporting, and the final export would be missing the pool. Calling it explicitly at the bottom means the provider reports while everything is still in place.&lt;/p&gt;&#xA;&lt;p&gt;Run it, send the same five requests, and press Ctrl+C:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;go run .&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.000222871 min=3.3e-07 max=0.00012808 buckets=&amp;#34;0.005=4&amp;#34; attrs.operation.name=get_order&#xA;msg=metric name=operation.duration unit=s kind=histogram count=1 sum=8.017e-05 min=8.017e-05 max=8.017e-05 buckets=&amp;#34;0.005=1&amp;#34; attrs.error.type=order_not_found attrs.operation.name=get_order&#xA;msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order&#xA;msg=metric name=db.client.operation.duration unit=s kind=histogram count=1 sum=6.488e-05 min=6.488e-05 max=6.488e-05 buckets=&amp;#34;0.005=1&amp;#34; attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite attrs.error.type=order_not_found&#xA;msg=metric name=db.client.operation.duration unit=s kind=histogram count=2 sum=0.00018016 min=8.186e-05 max=9.83e-05 buckets=&amp;#34;0.005=2&amp;#34; attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite&#xA;msg=metric name=phpboyscout.db.client.operation.active unit={operation} kind=sum value=0 monotonic=false attrs.db.namespace=orders attrs.db.operation.name=get_order&#xA;msg=metric name=db.client.connection.count unit={connection} kind=sum value=1 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=idle&#xA;msg=metric name=db.client.connection.count unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=used&#xA;msg=metric name=db.client.connection.max unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders&#xA;msg=metric name=phpboyscout.db.client.connection.wait_count unit={wait} kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders&#xA;msg=metric name=phpboyscout.db.client.connection.wait_time unit=s kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The &lt;code&gt;operation&lt;/code&gt; lines are as before. The new ones:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;db.client.operation.duration&lt;/code&gt;&lt;/strong&gt; has &lt;code&gt;count=2&lt;/code&gt; with no error and &lt;code&gt;count=1&lt;/code&gt; with &lt;code&gt;order_not_found&lt;/code&gt;. That&amp;rsquo;s three queries for five requests. We haven&amp;rsquo;t measured the cache yet, but we can already see it at work: two of the five requests never reached the database at all.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;db.client.connection.count&lt;/code&gt;&lt;/strong&gt; shows one idle connection and none in use, so the pool isn&amp;rsquo;t busy.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;db.client.connection.max&lt;/code&gt;&lt;/strong&gt; is &lt;code&gt;0&lt;/code&gt;, because we never set a limit on the pool, and 0 means unlimited.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;wait_count&lt;/code&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;wait_time&lt;/code&gt;&lt;/strong&gt; are both &lt;code&gt;0&lt;/code&gt;, so no request has queued for a connection. On a busy service, a number climbing here is the pool telling you it&amp;rsquo;s too small.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The names starting &lt;code&gt;phpboyscout.&lt;/code&gt; are metrics that OpenTelemetry&amp;rsquo;s standard naming (its &lt;em&gt;semantic conventions&lt;/em&gt;) doesn&amp;rsquo;t cover yet, so the module gives them its own prefix instead of inventing something that looks official.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-6-watch-the-cache&#34;&gt;Step 6: Watch the cache&#xA;&lt;/h2&gt;&lt;p&gt;The database numbers told us the cache is working, but only by inference. The &lt;code&gt;Cache&lt;/code&gt; helper measures it directly: hits, misses and how many entries it holds.&lt;/p&gt;&#xA;&lt;p&gt;The &lt;code&gt;service&lt;/code&gt; struct gets its last field:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;type service struct {&#xA;&#x9;conn  *sql.DB&#xA;&#x9;ops   *instrument.Operations&#xA;&#x9;db    *instrument.DB&#xA;&#x9;cache *instrument.Cache&#xA;&#xA;&#x9;mu        sync.Mutex&#xA;&#x9;customers map[int]string&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;customer&lt;/code&gt; records a hit when the map has the name, a miss when it doesn&amp;rsquo;t, and the map&amp;rsquo;s size each time it grows:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;func (s *service) customer(ctx context.Context, id int) (string, error) {&#xA;&#x9;s.mu.Lock()&#xA;&#x9;name, ok := s.customers[id]&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;if ok {&#xA;&#x9;&#x9;s.cache.Hit(ctx)&#xA;&#x9;&#x9;return name, nil&#xA;&#x9;}&#xA;&#xA;&#x9;s.cache.Miss(ctx)&#xA;&#xA;&#x9;err := s.db.Query(ctx, &amp;#34;get_order&amp;#34;, func() error {&#xA;&#x9;&#x9;err := s.conn.QueryRowContext(ctx, &amp;#34;SELECT customer FROM orders WHERE id = ?&amp;#34;, id).Scan(&amp;amp;name)&#xA;&#x9;&#x9;if errors.Is(err, sql.ErrNoRows) {&#xA;&#x9;&#x9;&#x9;return ErrNotFound&#xA;&#x9;&#x9;}&#xA;&#xA;&#x9;&#x9;return err&#xA;&#x9;})&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;return &amp;#34;&amp;#34;, err&#xA;&#x9;}&#xA;&#xA;&#x9;s.mu.Lock()&#xA;&#x9;s.customers[id] = name&#xA;&#x9;s.cache.SetSize(len(s.customers))&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;return name, nil&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;SetSize&lt;/code&gt; is called inside the lock, so the number it records matches the map at that moment.&lt;/p&gt;&#xA;&lt;p&gt;In &lt;code&gt;main&lt;/code&gt;, create the cache helper after the database helper, and add it to the &lt;code&gt;service&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;cache, err := instrument.NewCache(meter, &amp;#34;customers&amp;#34;)&#xA;if err != nil {&#xA;&#x9;log.Fatal(err)&#xA;}&#xA;defer cache.Close()&#xA;&#xA;s := &amp;amp;service{conn: conn, ops: ops, db: db, cache: cache, customers: map[int]string{}}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;&amp;quot;customers&amp;quot;&lt;/code&gt; is the cache&amp;rsquo;s name, and it becomes a label just like the others. &lt;code&gt;defer cache.Close()&lt;/code&gt; follows the pattern of the database helper, which is why &lt;code&gt;provider.Shutdown&lt;/code&gt; needs to stay explicit at the bottom.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-7-check-your-file&#34;&gt;Step 7: Check your file&#xA;&lt;/h2&gt;&lt;p&gt;That&amp;rsquo;s all the code. Your &lt;code&gt;main.go&lt;/code&gt; should now look like this, top to bottom:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;package main&#xA;&#xA;import (&#xA;&#x9;&amp;#34;context&amp;#34;&#xA;&#x9;&amp;#34;database/sql&amp;#34;&#xA;&#x9;&amp;#34;fmt&amp;#34;&#xA;&#x9;&amp;#34;log&amp;#34;&#xA;&#x9;&amp;#34;log/slog&amp;#34;&#xA;&#x9;&amp;#34;net/http&amp;#34;&#xA;&#x9;&amp;#34;os&amp;#34;&#xA;&#x9;&amp;#34;os/signal&amp;#34;&#xA;&#x9;&amp;#34;strconv&amp;#34;&#xA;&#x9;&amp;#34;sync&amp;#34;&#xA;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/errors&amp;#34;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/observability/instrument&amp;#34;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/observability/metrics&amp;#34;&#xA;&#x9;&amp;#34;gitlab.com/phpboyscout/go/observability/otelcore&amp;#34;&#xA;&#x9;_ &amp;#34;modernc.org/sqlite&amp;#34;&#xA;)&#xA;&#xA;// ErrNotFound has a kind, so a missing order is a queryable failure of its own.&#xA;var ErrNotFound = errors.NewSentinel(&amp;#34;order_not_found&amp;#34;, &amp;#34;no such order&amp;#34;)&#xA;&#xA;type service struct {&#xA;&#x9;conn  *sql.DB&#xA;&#x9;ops   *instrument.Operations&#xA;&#x9;db    *instrument.DB&#xA;&#x9;cache *instrument.Cache&#xA;&#xA;&#x9;mu        sync.Mutex&#xA;&#x9;customers map[int]string&#xA;}&#xA;&#xA;func (s *service) customer(ctx context.Context, id int) (string, error) {&#xA;&#x9;s.mu.Lock()&#xA;&#x9;name, ok := s.customers[id]&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;if ok {&#xA;&#x9;&#x9;s.cache.Hit(ctx)&#xA;&#x9;&#x9;return name, nil&#xA;&#x9;}&#xA;&#xA;&#x9;s.cache.Miss(ctx)&#xA;&#xA;&#x9;err := s.db.Query(ctx, &amp;#34;get_order&amp;#34;, func() error {&#xA;&#x9;&#x9;err := s.conn.QueryRowContext(ctx, &amp;#34;SELECT customer FROM orders WHERE id = ?&amp;#34;, id).Scan(&amp;amp;name)&#xA;&#x9;&#x9;if errors.Is(err, sql.ErrNoRows) {&#xA;&#x9;&#x9;&#x9;return ErrNotFound&#xA;&#x9;&#x9;}&#xA;&#xA;&#x9;&#x9;return err&#xA;&#x9;})&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;return &amp;#34;&amp;#34;, err&#xA;&#x9;}&#xA;&#xA;&#x9;s.mu.Lock()&#xA;&#x9;s.customers[id] = name&#xA;&#x9;s.cache.SetSize(len(s.customers))&#xA;&#x9;s.mu.Unlock()&#xA;&#xA;&#x9;return name, nil&#xA;}&#xA;&#xA;func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {&#xA;&#x9;id, err := strconv.Atoi(r.PathValue(&amp;#34;id&amp;#34;))&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;http.Error(w, &amp;#34;bad order ID&amp;#34;, http.StatusBadRequest)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;customer, err := instrument.ObserveResult(r.Context(), s.ops, &amp;#34;get_order&amp;#34;, func() (string, error) {&#xA;&#x9;&#x9;return s.customer(r.Context(), id)&#xA;&#x9;})&#xA;&#x9;if errors.Is(err, ErrNotFound) {&#xA;&#x9;&#x9;http.Error(w, err.Error(), http.StatusNotFound)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;http.Error(w, &amp;#34;internal error&amp;#34;, http.StatusInternalServerError)&#xA;&#x9;&#x9;return&#xA;&#x9;}&#xA;&#xA;&#x9;fmt.Fprintln(w, customer)&#xA;}&#xA;&#xA;func main() {&#xA;&#x9;ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)&#xA;&#x9;defer stop()&#xA;&#xA;&#x9;logger := slog.New(slog.NewTextHandler(os.Stdout, nil))&#xA;&#xA;&#x9;provider, err := metrics.NewProvider(ctx,&#xA;&#x9;&#x9;otelcore.Resource(&amp;#34;checkout&amp;#34;, &amp;#34;0.1.0&amp;#34;),&#xA;&#x9;&#x9;otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},&#xA;&#x9;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;meter := provider.Meter(&amp;#34;checkout&amp;#34;)&#xA;&#xA;&#x9;conn, err := sql.Open(&amp;#34;sqlite&amp;#34;, &amp;#34;file:checkout?mode=memory&amp;amp;cache=shared&amp;#34;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer conn.Close()&#xA;&#xA;&#x9;_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);&#xA;&#x9;&#x9;INSERT INTO orders VALUES (1, &amp;#39;Ada&amp;#39;), (2, &amp;#39;Grace&amp;#39;);`)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;ops, err := instrument.NewOperations(meter)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;db, err := instrument.NewDB(meter, &amp;#34;orders&amp;#34;, instrument.WithDBSystem(&amp;#34;sqlite&amp;#34;))&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer db.Close()&#xA;&#xA;&#x9;err = db.RegisterPool(func() instrument.PoolStats {&#xA;&#x9;&#x9;stats := conn.Stats()&#xA;&#xA;&#x9;&#x9;return instrument.PoolStats{&#xA;&#x9;&#x9;&#x9;MaxOpen:     stats.MaxOpenConnections,&#xA;&#x9;&#x9;&#x9;Open:        stats.OpenConnections,&#xA;&#x9;&#x9;&#x9;InUse:       stats.InUse,&#xA;&#x9;&#x9;&#x9;Idle:        stats.Idle,&#xA;&#x9;&#x9;&#x9;WaitCount:   stats.WaitCount,&#xA;&#x9;&#x9;&#x9;WaitSeconds: stats.WaitDuration.Seconds(),&#xA;&#x9;&#x9;}&#xA;&#x9;})&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#xA;&#x9;cache, err := instrument.NewCache(meter, &amp;#34;customers&amp;#34;)&#xA;&#x9;if err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;&#x9;defer cache.Close()&#xA;&#xA;&#x9;s := &amp;amp;service{conn: conn, ops: ops, db: db, cache: cache, customers: map[int]string{}}&#xA;&#xA;&#x9;mux := http.NewServeMux()&#xA;&#x9;mux.HandleFunc(&amp;#34;GET /orders/{id}&amp;#34;, s.getOrder)&#xA;&#xA;&#x9;server := &amp;amp;http.Server{Addr: &amp;#34;:8080&amp;#34;, Handler: mux}&#xA;&#xA;&#x9;go func() {&#xA;&#x9;&#x9;if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {&#xA;&#x9;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;&#x9;}&#xA;&#x9;}()&#xA;&#xA;&#x9;log.Println(&amp;#34;listening on :8080&amp;#34;)&#xA;&#x9;&amp;lt;-ctx.Done()&#xA;&#xA;&#x9;_ = server.Shutdown(context.Background())&#xA;&#xA;&#x9;if err := provider.Shutdown(context.Background()); err != nil {&#xA;&#x9;&#x9;log.Fatal(err)&#xA;&#x9;}&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;If yours differs, this is a good moment to fix it before running, since the next step reads the whole output.&lt;/p&gt;&#xA;&lt;h2 id=&#34;step-8-send-five-requests-and-read-the-whole-story&#34;&gt;Step 8: Send five requests and read the whole story&#xA;&lt;/h2&gt;&lt;p&gt;Run it one last time:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;go run .&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Send the same five requests from the second terminal:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Then press Ctrl+C in the first terminal. You&amp;rsquo;ll get all the metrics the service now has. Your durations will differ, and the lines may come out in a slightly different order, but the counts should match:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; class=&#34;chroma&#34;&gt;&lt;code class=&#34;language-text&#34; data-lang=&#34;text&#34;&gt;msg=metric name=operation.duration unit=s kind=histogram count=1 sum=5.86e-05 min=5.86e-05 max=5.86e-05 buckets=&amp;#34;0.005=1&amp;#34; attrs.error.type=order_not_found attrs.operation.name=get_order&#xA;msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.00024010100000000002 min=1.86e-06 max=0.000119371 buckets=&amp;#34;0.005=4&amp;#34; attrs.operation.name=get_order&#xA;msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order&#xA;msg=metric name=db.client.operation.duration unit=s kind=histogram count=1 sum=4.262e-05 min=4.262e-05 max=4.262e-05 buckets=&amp;#34;0.005=1&amp;#34; attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite attrs.error.type=order_not_found&#xA;msg=metric name=db.client.operation.duration unit=s kind=histogram count=2 sum=0.000187021 min=9.177e-05 max=9.5251e-05 buckets=&amp;#34;0.005=2&amp;#34; attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite&#xA;msg=metric name=phpboyscout.db.client.operation.active unit={operation} kind=sum value=0 monotonic=false attrs.db.namespace=orders attrs.db.operation.name=get_order&#xA;msg=metric name=db.client.connection.count unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=used&#xA;msg=metric name=db.client.connection.count unit={connection} kind=sum value=1 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=idle&#xA;msg=metric name=db.client.connection.max unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders&#xA;msg=metric name=phpboyscout.db.client.connection.wait_count unit={wait} kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders&#xA;msg=metric name=phpboyscout.db.client.connection.wait_time unit=s kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders&#xA;msg=metric name=cache.operations unit={operation} kind=sum value=3 monotonic=true attrs.cache.name=customers attrs.cache.result=miss&#xA;msg=metric name=cache.operations unit={operation} kind=sum value=2 monotonic=true attrs.cache.name=customers attrs.cache.result=hit&#xA;msg=metric name=cache.size unit={entry} kind=gauge value=2 attrs.cache.name=customers&#xA;msg=metric name=cache.capacity unit={entry} kind=gauge value=0 attrs.cache.name=customers&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;If the service ran for more than a minute, you&amp;rsquo;ll also see an earlier block from a periodic report. The values are running totals, so read the last block.&lt;/p&gt;&#xA;&lt;p&gt;Put together, those lines tell the story of the five requests:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Five requests, one failure.&lt;/strong&gt; &lt;code&gt;operation.duration&lt;/code&gt; counts 4 that worked and 1 labelled &lt;code&gt;order_not_found&lt;/code&gt;, all finishing within 5 milliseconds.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Two hits, three misses.&lt;/strong&gt; &lt;code&gt;cache.operations&lt;/code&gt; shows the map already held the name twice: the second and third requests for order 1.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Three queries.&lt;/strong&gt; &lt;code&gt;db.client.operation.duration&lt;/code&gt; counts 2 queries that found an order and 1 that didn&amp;rsquo;t. Three queries for three misses&amp;hellip; the database only saw what the cache couldn&amp;rsquo;t answer.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Two entries.&lt;/strong&gt; &lt;code&gt;cache.size&lt;/code&gt; is 2, for orders 1 and 2. Order 99 wasn&amp;rsquo;t stored, because the database had no customer to give it. &lt;code&gt;cache.capacity&lt;/code&gt; is 0 because we never set one, which means unbounded.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A quiet pool.&lt;/strong&gt; One idle connection, none in use, nothing waiting.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;These are the OpenTelemetry names a collector would receive. The &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/reference/metric-names/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;metric names reference&lt;/a&gt; lists them all, and &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/reference/slog-records/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Slog record fields&lt;/a&gt; explains each field on the line.&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-we-built&#34;&gt;What we built&#xA;&lt;/h2&gt;&lt;p&gt;We started with a service that answered &lt;code&gt;curl&lt;/code&gt; and said nothing else. It now reports:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;how many requests it served, how many failed and &lt;em&gt;why&lt;/em&gt; (by error kind, not message), and how long they took;&lt;/li&gt;&#xA;&lt;li&gt;how many of those requests reached the database, and how long the queries took;&lt;/li&gt;&#xA;&lt;li&gt;whether the connection pool is busy, and whether anything is waiting for it;&lt;/li&gt;&#xA;&lt;li&gt;how often the cache can skip the database, and how big it has grown.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;And the code that does the actual work barely changed. The handler still returns the responses it always did, errors still come back untouched, and the instrumentation sits around the work rather than inside it: one wrapper round the request, one round the query, and a hit or a miss where the map is checked.&lt;/p&gt;&#xA;&lt;h2 id=&#34;where-to-go-from-here&#34;&gt;Where to go from here&#xA;&lt;/h2&gt;&lt;p&gt;We used &lt;code&gt;otelcore.Slog&lt;/code&gt; so the output could be read by eye, but the &lt;code&gt;instrument&lt;/code&gt; code doesn&amp;rsquo;t depend on it. Give the helpers a meter from any provider and they behave the same way.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;To send these metrics to a collector instead, change the &lt;code&gt;Exporter&lt;/code&gt; to &lt;code&gt;otelcore.Collector{Endpoint: &amp;quot;http://localhost:4318&amp;quot;}&lt;/code&gt; and follow &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/tutorials/export-all-three-signals/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Export all three signals to a collector you can watch&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;To see metrics sooner than once a minute, or write them to a file, read &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/how-to/send-telemetry-to-the-terminal-or-a-file/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Send telemetry to the terminal or a file&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;To instrument a &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/how-to/instrument-a-circuit-breaker/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;circuit breaker&lt;/a&gt; or a &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/how-to/instrument-a-queue/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;queue&lt;/a&gt;, the how-to guides follow the pattern you used for the database and the cache.&lt;/li&gt;&#xA;&lt;li&gt;To understand the signals properly, and why label values have to stay bounded, read &lt;a class=&#34;link&#34; href=&#34;https://observability.go.phpboyscout.uk/explanation/red-and-use/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;RED, USE and cardinality&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;</description>
        </item></channel>
</rss>
