Featured image of post Five requests, three queries

Five requests, three queries

Build a small Go service, then teach it to tell you how many requests it served, how many failed, how long they took and whether its cache is pulling its weight, one stage at a time.

A small service will happily answer curl all day without telling you much about how it got there. You can see that it replied. What you can’t see is whether the cache in front of the database is doing anything at all, how long the query took, or how many of those requests failed without a word.

Metrics are how you find out, and the catch is that a raw OpenTelemetry meter hands you a counter and leaves the names, the units and the labels up to you… and then you get to make all of those decisions again in the next service, slightly differently.

The instrument package in go/observability comes with those decisions already made, and this walkthrough is a gentle tour of it. We’ll build a small service from scratch, then add metrics to it a stage at a time, running it after each stage so you can see what each new piece reports before moving on to the next.

What we’re going to do

The service is about as small as a real one gets. It answers GET /orders/{id} with the name of the customer who placed that order. The orders live in a SQLite database, and the service keeps each customer name it has already looked up in a map (a very simple cache), so asking for one order twice only bothers the database once.

Over the course of the tutorial we’ll:

  1. Build the service with no metrics at all, and check it works.
  2. Teach it to stop cleanly when you press Ctrl+C, because that’s when it will print its metrics.
  3. Give it a meter and measure each request: how many there were, how many failed and how long they took.
  4. Watch the database: how many queries reached it, and whether its connection pool is under pressure.
  5. Watch the cache: how often the map already held the name.
  6. Send it five requests and read the whole story it tells about them.

Each stage builds on the last, and the service compiles and runs at the end of each one, so if something goes wrong you’ll know which stage it went wrong in.

A few words before we start

If you’ve not worked with metrics before, here’s the vocabulary we’ll lean on. Feel free to skip this if it’s familiar.

A metric is a number your program keeps up to date as it runs, like a car’s mileage counter, rather than a line written for each event the way a log is. Once in a while the program reports the current numbers somewhere. That’s cheap, it doesn’t grow with traffic, and it’s pretty good at questions like “how many?” and “how long, usually?”.

OpenTelemetry (often shortened to OTel) is the open standard most tools now use for metrics, traces and logs. It’s worth a search if the name is new to you. In OTel terms you’ll meet three things in this tutorial:

  • a provider, which owns all the metrics and decides when and where they’re reported;
  • a meter, the handle you ask for from the provider and create your metrics from;
  • an exporter, which is where the provider sends them. Normally that’s a collector over the network. Here it’ll be your terminal.

Finally, two acronyms that give the instrument package its shape. RED (rate, errors, duration) is what you measure about work: how often it happens, how often it fails and how long it takes. USE (utilisation, saturation, errors) is what you measure about resources, such as a connection pool: how busy it is, and whether anything is queueing for it. The module’s own explanation page goes into both properly.

What you’ll need

  • Go at the version on the go line of go/observability’s go.mod, or later. That moves as the module does, so check it rather than trusting a number here.
  • curl.
  • Two terminals: one to run the service in, one to send it requests.
  • About forty minutes, a bit more if you stop to read the output carefully (which is rather the point).

You don’t need a collector, Docker, a dashboard or go-tool-base. It’s a plain net/http service.

This post is a slower, more explained version of the canonical tutorial in the go/observability docs, which is the copy kept up to date as the module moves: https://observability.go.phpboyscout.uk/tutorials/instrument-a-small-service/. Each stage below was run end to end against go/observability v0.6.0 and go/errors v0.4.0 on Go 1.27.3, and the output shown is what came back.

If you followed part 6 of the web service series, you’ve already seen traces, which follow one request through each hop it makes. Metrics come at it from the other side: they tell you about all the requests added up, which is what you want when the question is “is the cache earning its memory?” rather than “why was that one request slow?”.

Step 1: Create the project

Make a directory for the service, start a Go module in it, and fetch the three libraries we’ll use:

mkdir checkout && cd checkout
go mod init example.com/checkout
go get gitlab.com/phpboyscout/go/observability
go get gitlab.com/phpboyscout/go/errors
go get modernc.org/sqlite
  • go/observability holds the instrument package and the plumbing to set up a provider.
  • go/errors lets us give an error a kind, a short fixed name for a type of failure. That turns out to matter for metrics, and we’ll see why in Step 4.
  • modernc.org/sqlite is SQLite written in pure Go, so you don’t need a C compiler. It’s a big package, and the first build can take a minute or so.

Step 2: Build a service that works and says nothing

Before measuring anything, we want a service that works, so that when the numbers start appearing we know what they should say. Create main.go with this in it:

package main

import (
	"context"
	"database/sql"
	"fmt"
	"log"
	"net/http"
	"strconv"
	"sync"

	"gitlab.com/phpboyscout/go/errors"
	_ "modernc.org/sqlite"
)

// ErrNotFound has a kind, so a missing order is a queryable failure of its own.
var ErrNotFound = errors.NewSentinel("order_not_found", "no such order")

type service struct {
	conn *sql.DB

	mu        sync.Mutex
	customers map[int]string
}

func (s *service) customer(ctx context.Context, id int) (string, error) {
	s.mu.Lock()
	name, ok := s.customers[id]
	s.mu.Unlock()

	if ok {
		return name, nil
	}

	err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
	if errors.Is(err, sql.ErrNoRows) {
		return "", ErrNotFound
	}

	if err != nil {
		return "", err
	}

	s.mu.Lock()
	s.customers[id] = name
	s.mu.Unlock()

	return name, nil
}

func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
	id, err := strconv.Atoi(r.PathValue("id"))
	if err != nil {
		http.Error(w, "bad order ID", http.StatusBadRequest)
		return
	}

	customer, err := s.customer(r.Context(), id)
	if errors.Is(err, ErrNotFound) {
		http.Error(w, err.Error(), http.StatusNotFound)
		return
	}

	if err != nil {
		http.Error(w, "internal error", http.StatusInternalServerError)
		return
	}

	fmt.Fprintln(w, customer)
}

func main() {
	conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()

	_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
		INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
	if err != nil {
		log.Fatal(err)
	}

	s := &service{conn: conn, customers: map[int]string{}}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /orders/{id}", s.getOrder)

	log.Println("listening on :8080")
	log.Fatal(http.ListenAndServe(":8080", mux))
}

That’s a fair bit of code in one go, so let’s walk through it from the top.

  • ErrNotFound is the error we return when an order doesn’t exist. errors.NewSentinel gives it the kind order_not_found as well as a message. For now it’s just an error; once metrics arrive, that kind becomes the label that tells “not found” apart from any other failure.
  • service holds the database connection and the cache. The cache is a plain map from order ID to customer name. Two requests can arrive at once, so the map sits behind a mutex (mu) to stop them treading on each other.
  • customer is where the work happens. It checks the map first and returns straight away if the name is there. If it isn’t, it asks the database, turns SQL’s “no rows” error into our ErrNotFound, and stores the name in the map for next time.
  • getOrder is the HTTP handler. It reads the ID from the URL, calls customer, and turns the result into a response: the name, a 404 for an unknown order, or a 500 for anything else.
  • main opens the database, creates the orders table with two orders in it, registers the handler and starts listening on port 8080.

The database is held in memory, so it vanishes when the program stops, which is fine for a tutorial. The cache=shared part of the connection string makes all the connections in Go’s pool share one in-memory database, rather than each getting its own empty one.

Run it, and leave it running:

go run .
2026/11/13 12:25:48 listening on :8080

In your second terminal, ask for an order that exists and one that doesn’t:

curl localhost:8080/orders/1
curl localhost:8080/orders/99
Ada
no such order

If you see address already in use, something else already has port 8080 (it did on my machine). Change the port in main.go, and in the curl commands from here on.

Stop the service with Ctrl+C. It works, and it tells us nothing about how.

Step 3: Teach the service to stop cleanly

This step doesn’t add any metrics yet, but the metrics depend on it, so it’s worth doing on its own.

The provider we’ll add in the next step reports its numbers every 60 seconds, and once more when it’s shut down. For a tutorial, that last report is what we want to read: send some requests, press Ctrl+C, and see the totals.

Right now, though, Ctrl+C kills the program hard. Go’s default response to that signal is to exit immediately, so none of the clean-up gets a chance to run. A request that’s halfway through is cut off mid-reply. The defer conn.Close() near the top of main is skipped, and the database connection is simply abandoned. With an in-memory database in a tutorial that costs us very little, but a real service treated like this loses work every time it’s restarted, and in the next step it would also lose the provider’s last report before it’s ever written.

The root of it is that http.ListenAndServe only does half of a server’s job. It starts listening and serving requests, and then it blocks for as long as the program lives. There’s no way to ask it to stop, so the only way out is for the process to die. And even if it did return, log.Fatal exits on the spot, which skips the deferred calls too.

So in this step we give the server its other half. main will notice Ctrl+C, ask the HTTP server to stop politely, wait for it to finish, and then carry on to the end of the function, so that everything deferred gets closed in order.

Add "os" and "os/signal" to the imports, so the block reads:

import (
	"context"
	"database/sql"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"sync"

	"gitlab.com/phpboyscout/go/errors"
	_ "modernc.org/sqlite"
)

Then replace main with this version:

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()

	_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
		INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
	if err != nil {
		log.Fatal(err)
	}

	s := &service{conn: conn, customers: map[int]string{}}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /orders/{id}", s.getOrder)

	server := &http.Server{Addr: ":8080", Handler: mux}

	go func() {
		if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	log.Println("listening on :8080")
	<-ctx.Done()

	_ = server.Shutdown(context.Background())
}

Three things changed:

  • signal.NotifyContext gives us a context that’s cancelled when you press Ctrl+C, instead of the program being killed outright.
  • The server now runs in its own goroutine, so main is free to wait. <-ctx.Done() is that wait: it blocks until Ctrl+C.
  • When it comes, server.Shutdown stops accepting new requests and lets any in-flight ones finish. ListenAndServe then returns http.ErrServerClosed, which is the normal “I was asked to stop” answer, so the goroutine only treats any other error as fatal.

Run it again, send a request if you like, and press Ctrl+C:

go run .
2026/11/13 12:26:10 listening on :8080

This time the program returns you to the prompt without any error. It looks no different, but main now reaches its last line, and that’s where the next step puts the metrics report.

Step 4: Give it a meter and measure each request

Now the first real metrics. We’ll set up a provider, get a meter from it, and use the instrument package’s Operations helper to measure the request.

An operation is just a piece of work you care about. Here it’s “serve an order”. Operations creates two metrics for it:

  • operation.duration, a histogram of how long each operation took. A histogram doesn’t keep each individual timing. It keeps a count, a total, the fastest and slowest, and how many fell into each of a set of time buckets. The count gives you the rate, and counting only the samples labelled with an error gives you the errors, so this one metric covers all three letters of RED.
  • operation.active, how many operations are running at this moment.

This step touches the file in four places, top to bottom. First the imports: add log/slog and the three go/observability packages.

import (
	"context"
	"database/sql"
	"fmt"
	"log"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"sync"

	"gitlab.com/phpboyscout/go/errors"
	"gitlab.com/phpboyscout/go/observability/instrument"
	"gitlab.com/phpboyscout/go/observability/metrics"
	"gitlab.com/phpboyscout/go/observability/otelcore"
	_ "modernc.org/sqlite"
)

Next, the service struct gets a field to hold the operations helper:

type service struct {
	conn *sql.DB
	ops  *instrument.Operations

	mu        sync.Mutex
	customers map[int]string
}

Then getOrder wraps its call to customer in instrument.ObserveResult. Here’s the whole handler:

func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
	id, err := strconv.Atoi(r.PathValue("id"))
	if err != nil {
		http.Error(w, "bad order ID", http.StatusBadRequest)
		return
	}

	customer, err := instrument.ObserveResult(r.Context(), s.ops, "get_order", func() (string, error) {
		return s.customer(r.Context(), id)
	})
	if errors.Is(err, ErrNotFound) {
		http.Error(w, err.Error(), http.StatusNotFound)
		return
	}

	if err != nil {
		http.Error(w, "internal error", http.StatusInternalServerError)
		return
	}

	fmt.Fprintln(w, customer)
}

ObserveResult runs the function you give it, times it, and notes what kind of error came back (if any). Then it hands you the value and the error just as your function returned them, so the rest of the handler carries on unchanged. (It’s called ObserveResult because it passes a value back. For work that only returns an error there’s a plain Observe.)

"get_order" is the operation’s name, and it becomes a label on each measurement. Labels are how you slice a metric later (“show me get_order, failures only”), but each distinct label value creates a separate series that whatever receives your metrics has to store. So names like this come from a small fixed set that you write in the code. Don’t build one from the order ID or anything else a user controls, or a busy day can turn into one series per order.

Finally, main. It now builds the provider at the top and shuts it down at the bottom:

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

	provider, err := metrics.NewProvider(ctx,
		otelcore.Resource("checkout", "0.1.0"),
		otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
	)
	if err != nil {
		log.Fatal(err)
	}

	meter := provider.Meter("checkout")

	conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()

	_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
		INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
	if err != nil {
		log.Fatal(err)
	}

	ops, err := instrument.NewOperations(meter)
	if err != nil {
		log.Fatal(err)
	}

	s := &service{conn: conn, ops: ops, customers: map[int]string{}}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /orders/{id}", s.getOrder)

	server := &http.Server{Addr: ":8080", Handler: mux}

	go func() {
		if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	log.Println("listening on :8080")
	<-ctx.Done()

	_ = server.Shutdown(context.Background())

	if err := provider.Shutdown(context.Background()); err != nil {
		log.Fatal(err)
	}
}

Going through the new parts in order:

  • The logger. A standard slog logger writing plain text to the terminal.
  • The provider. metrics.NewProvider builds it. otelcore.Resource("checkout", "0.1.0") names the service and its version, so whatever receives the metrics knows who sent them. otelcore.Settings switches metrics on and picks the exporter, and otelcore.Slog(logger) is the exporter that writes each metric as a log line instead of sending it over the network.
  • The meter. provider.Meter("checkout") is what the instrument helpers create their metrics on.
  • The operations helper. instrument.NewOperations(meter) creates the two metrics, and goes into the service.
  • provider.Shutdown. At the very end, after the HTTP server has stopped, this makes the provider send its final report. That’s the export we’ll be reading.

Run it:

go run .

This time, in the second terminal, send it five requests: order 1 three times, order 2 once, and order 99, which doesn’t exist.

for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done

Go back to the first terminal and press Ctrl+C. The provider’s last report appears, one line per metric. Each line starts with time= and level=INFO, which I’ve left off here so the rest fits:

msg=metric name=operation.duration unit=s kind=histogram count=1 sum=6.565e-05 min=6.565e-05 max=6.565e-05 buckets="0.005=1" attrs.error.type=order_not_found attrs.operation.name=get_order
msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.00021654999999999998 min=3.1e-07 max=0.00012721 buckets="0.005=4" attrs.operation.name=get_order
msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order

Here’s how to read it.

The two operation.duration lines are one metric with two different labels. The line with attrs.error.type=order_not_found has count=1: that’s the request for order 99. The line with no error label has count=4: the other four. One plus four is all five requests, so you have the rate and the errors straight away.

The duration lives in the other fields. sum is the total time in seconds, min and max are the fastest and slowest, and buckets="0.005=4" says all four successful requests finished within 5 milliseconds. Your timings will differ (and the odd long string of 9s in a sum is ordinary floating-point arithmetic), but the counts should match.

Notice the error label says order_not_found, the kind we gave ErrNotFound back in Step 2. That’s why it was worth giving the error a kind. An error without one is labelled _OTHER, which is accurate and not much help when you’re trying to tell one failure from another. You can’t label with the error’s message instead, because messages often contain IDs and other per-request values, and each one would become a new series.

operation.active is 0, because no request was running at the moment the provider reported. Under real load, that’s the number that tells you how much is in flight.

Step 5: Watch the database

We now know how long requests take, but not where the time goes. The next question is what the database is doing. The instrument package’s DB helper measures two things:

  • Each query (RED again): how many, how many failed, how long they took.
  • The connection pool (USE): how many connections are open, how many are busy, and whether anything has had to wait for one.

The imports don’t change this time, since instrument is already there. The service struct gets a db field:

type service struct {
	conn *sql.DB
	ops  *instrument.Operations
	db   *instrument.DB

	mu        sync.Mutex
	customers map[int]string
}

customer wraps its query in s.db.Query:

func (s *service) customer(ctx context.Context, id int) (string, error) {
	s.mu.Lock()
	name, ok := s.customers[id]
	s.mu.Unlock()

	if ok {
		return name, nil
	}

	err := s.db.Query(ctx, "get_order", func() error {
		err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
		if errors.Is(err, sql.ErrNoRows) {
			return ErrNotFound
		}

		return err
	})
	if err != nil {
		return "", err
	}

	s.mu.Lock()
	s.customers[id] = name
	s.mu.Unlock()

	return name, nil
}

It works like ObserveResult: give it a name and a function, and it times the function and records its error. The query name becomes a label as well, so the fixed-names rule from Step 4 applies: "get_order" is fine, the SQL text with an ID in it isn’t.

Look closely and you’ll see the sql.ErrNoRows check has moved inside the function. That’s so the helper sees ErrNotFound, with its order_not_found kind, instead of SQL’s own error, which has no kind and would be recorded as _OTHER.

And main gains the database helper, between the operations helper and the service:

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

	provider, err := metrics.NewProvider(ctx,
		otelcore.Resource("checkout", "0.1.0"),
		otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
	)
	if err != nil {
		log.Fatal(err)
	}

	meter := provider.Meter("checkout")

	conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()

	_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
		INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
	if err != nil {
		log.Fatal(err)
	}

	ops, err := instrument.NewOperations(meter)
	if err != nil {
		log.Fatal(err)
	}

	db, err := instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite"))
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	err = db.RegisterPool(func() instrument.PoolStats {
		stats := conn.Stats()

		return instrument.PoolStats{
			MaxOpen:     stats.MaxOpenConnections,
			Open:        stats.OpenConnections,
			InUse:       stats.InUse,
			Idle:        stats.Idle,
			WaitCount:   stats.WaitCount,
			WaitSeconds: stats.WaitDuration.Seconds(),
		}
	})
	if err != nil {
		log.Fatal(err)
	}

	s := &service{conn: conn, ops: ops, db: db, customers: map[int]string{}}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /orders/{id}", s.getOrder)

	server := &http.Server{Addr: ":8080", Handler: mux}

	go func() {
		if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	log.Println("listening on :8080")
	<-ctx.Done()

	_ = server.Shutdown(context.Background())

	if err := provider.Shutdown(context.Background()); err != nil {
		log.Fatal(err)
	}
}

instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite")) creates the query metrics for a database we’re calling orders, and records that it’s SQLite.

db.RegisterPool is how the helper finds out about the pool. The function you pass it is called each time the provider reports, and it copies Go’s own pool statistics (conn.Stats(), from database/sql) into the shape the helper expects. Keep it cheap, because it runs on every report.

You might have wondered why provider.Shutdown sits at the end of main instead of being deferred. defer db.Close() stops the pool reporting, and deferred calls run when main returns. Had we written defer provider.Shutdown(...) up at the top, Go would run it last (deferred calls run in reverse order), after the pool had already stopped reporting, and the final export would be missing the pool. Calling it explicitly at the bottom means the provider reports while everything is still in place.

Run it, send the same five requests, and press Ctrl+C:

go run .
for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done
msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.000222871 min=3.3e-07 max=0.00012808 buckets="0.005=4" attrs.operation.name=get_order
msg=metric name=operation.duration unit=s kind=histogram count=1 sum=8.017e-05 min=8.017e-05 max=8.017e-05 buckets="0.005=1" attrs.error.type=order_not_found attrs.operation.name=get_order
msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order
msg=metric name=db.client.operation.duration unit=s kind=histogram count=1 sum=6.488e-05 min=6.488e-05 max=6.488e-05 buckets="0.005=1" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite attrs.error.type=order_not_found
msg=metric name=db.client.operation.duration unit=s kind=histogram count=2 sum=0.00018016 min=8.186e-05 max=9.83e-05 buckets="0.005=2" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite
msg=metric name=phpboyscout.db.client.operation.active unit={operation} kind=sum value=0 monotonic=false attrs.db.namespace=orders attrs.db.operation.name=get_order
msg=metric name=db.client.connection.count unit={connection} kind=sum value=1 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=idle
msg=metric name=db.client.connection.count unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=used
msg=metric name=db.client.connection.max unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_count unit={wait} kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_time unit=s kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders

The operation lines are as before. The new ones:

  • db.client.operation.duration has count=2 with no error and count=1 with order_not_found. That’s three queries for five requests. We haven’t measured the cache yet, but we can already see it at work: two of the five requests never reached the database at all.
  • db.client.connection.count shows one idle connection and none in use, so the pool isn’t busy.
  • db.client.connection.max is 0, because we never set a limit on the pool, and 0 means unlimited.
  • wait_count and wait_time are both 0, so no request has queued for a connection. On a busy service, a number climbing here is the pool telling you it’s too small.

The names starting phpboyscout. are metrics that OpenTelemetry’s standard naming (its semantic conventions) doesn’t cover yet, so the module gives them its own prefix instead of inventing something that looks official.

Step 6: Watch the cache

The database numbers told us the cache is working, but only by inference. The Cache helper measures it directly: hits, misses and how many entries it holds.

The service struct gets its last field:

type service struct {
	conn  *sql.DB
	ops   *instrument.Operations
	db    *instrument.DB
	cache *instrument.Cache

	mu        sync.Mutex
	customers map[int]string
}

customer records a hit when the map has the name, a miss when it doesn’t, and the map’s size each time it grows:

func (s *service) customer(ctx context.Context, id int) (string, error) {
	s.mu.Lock()
	name, ok := s.customers[id]
	s.mu.Unlock()

	if ok {
		s.cache.Hit(ctx)
		return name, nil
	}

	s.cache.Miss(ctx)

	err := s.db.Query(ctx, "get_order", func() error {
		err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
		if errors.Is(err, sql.ErrNoRows) {
			return ErrNotFound
		}

		return err
	})
	if err != nil {
		return "", err
	}

	s.mu.Lock()
	s.customers[id] = name
	s.cache.SetSize(len(s.customers))
	s.mu.Unlock()

	return name, nil
}

SetSize is called inside the lock, so the number it records matches the map at that moment.

In main, create the cache helper after the database helper, and add it to the service:

cache, err := instrument.NewCache(meter, "customers")
if err != nil {
	log.Fatal(err)
}
defer cache.Close()

s := &service{conn: conn, ops: ops, db: db, cache: cache, customers: map[int]string{}}

"customers" is the cache’s name, and it becomes a label just like the others. defer cache.Close() follows the pattern of the database helper, which is why provider.Shutdown needs to stay explicit at the bottom.

Step 7: Check your file

That’s all the code. Your main.go should now look like this, top to bottom:

package main

import (
	"context"
	"database/sql"
	"fmt"
	"log"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"sync"

	"gitlab.com/phpboyscout/go/errors"
	"gitlab.com/phpboyscout/go/observability/instrument"
	"gitlab.com/phpboyscout/go/observability/metrics"
	"gitlab.com/phpboyscout/go/observability/otelcore"
	_ "modernc.org/sqlite"
)

// ErrNotFound has a kind, so a missing order is a queryable failure of its own.
var ErrNotFound = errors.NewSentinel("order_not_found", "no such order")

type service struct {
	conn  *sql.DB
	ops   *instrument.Operations
	db    *instrument.DB
	cache *instrument.Cache

	mu        sync.Mutex
	customers map[int]string
}

func (s *service) customer(ctx context.Context, id int) (string, error) {
	s.mu.Lock()
	name, ok := s.customers[id]
	s.mu.Unlock()

	if ok {
		s.cache.Hit(ctx)
		return name, nil
	}

	s.cache.Miss(ctx)

	err := s.db.Query(ctx, "get_order", func() error {
		err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
		if errors.Is(err, sql.ErrNoRows) {
			return ErrNotFound
		}

		return err
	})
	if err != nil {
		return "", err
	}

	s.mu.Lock()
	s.customers[id] = name
	s.cache.SetSize(len(s.customers))
	s.mu.Unlock()

	return name, nil
}

func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
	id, err := strconv.Atoi(r.PathValue("id"))
	if err != nil {
		http.Error(w, "bad order ID", http.StatusBadRequest)
		return
	}

	customer, err := instrument.ObserveResult(r.Context(), s.ops, "get_order", func() (string, error) {
		return s.customer(r.Context(), id)
	})
	if errors.Is(err, ErrNotFound) {
		http.Error(w, err.Error(), http.StatusNotFound)
		return
	}

	if err != nil {
		http.Error(w, "internal error", http.StatusInternalServerError)
		return
	}

	fmt.Fprintln(w, customer)
}

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

	provider, err := metrics.NewProvider(ctx,
		otelcore.Resource("checkout", "0.1.0"),
		otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
	)
	if err != nil {
		log.Fatal(err)
	}

	meter := provider.Meter("checkout")

	conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()

	_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
		INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
	if err != nil {
		log.Fatal(err)
	}

	ops, err := instrument.NewOperations(meter)
	if err != nil {
		log.Fatal(err)
	}

	db, err := instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite"))
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	err = db.RegisterPool(func() instrument.PoolStats {
		stats := conn.Stats()

		return instrument.PoolStats{
			MaxOpen:     stats.MaxOpenConnections,
			Open:        stats.OpenConnections,
			InUse:       stats.InUse,
			Idle:        stats.Idle,
			WaitCount:   stats.WaitCount,
			WaitSeconds: stats.WaitDuration.Seconds(),
		}
	})
	if err != nil {
		log.Fatal(err)
	}

	cache, err := instrument.NewCache(meter, "customers")
	if err != nil {
		log.Fatal(err)
	}
	defer cache.Close()

	s := &service{conn: conn, ops: ops, db: db, cache: cache, customers: map[int]string{}}

	mux := http.NewServeMux()
	mux.HandleFunc("GET /orders/{id}", s.getOrder)

	server := &http.Server{Addr: ":8080", Handler: mux}

	go func() {
		if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	log.Println("listening on :8080")
	<-ctx.Done()

	_ = server.Shutdown(context.Background())

	if err := provider.Shutdown(context.Background()); err != nil {
		log.Fatal(err)
	}
}

If yours differs, this is a good moment to fix it before running, since the next step reads the whole output.

Step 8: Send five requests and read the whole story

Run it one last time:

go run .

Send the same five requests from the second terminal:

for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done

Then press Ctrl+C in the first terminal. You’ll get all the metrics the service now has. Your durations will differ, and the lines may come out in a slightly different order, but the counts should match:

msg=metric name=operation.duration unit=s kind=histogram count=1 sum=5.86e-05 min=5.86e-05 max=5.86e-05 buckets="0.005=1" attrs.error.type=order_not_found attrs.operation.name=get_order
msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.00024010100000000002 min=1.86e-06 max=0.000119371 buckets="0.005=4" attrs.operation.name=get_order
msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order
msg=metric name=db.client.operation.duration unit=s kind=histogram count=1 sum=4.262e-05 min=4.262e-05 max=4.262e-05 buckets="0.005=1" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite attrs.error.type=order_not_found
msg=metric name=db.client.operation.duration unit=s kind=histogram count=2 sum=0.000187021 min=9.177e-05 max=9.5251e-05 buckets="0.005=2" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite
msg=metric name=phpboyscout.db.client.operation.active unit={operation} kind=sum value=0 monotonic=false attrs.db.namespace=orders attrs.db.operation.name=get_order
msg=metric name=db.client.connection.count unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=used
msg=metric name=db.client.connection.count unit={connection} kind=sum value=1 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=idle
msg=metric name=db.client.connection.max unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_count unit={wait} kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_time unit=s kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=cache.operations unit={operation} kind=sum value=3 monotonic=true attrs.cache.name=customers attrs.cache.result=miss
msg=metric name=cache.operations unit={operation} kind=sum value=2 monotonic=true attrs.cache.name=customers attrs.cache.result=hit
msg=metric name=cache.size unit={entry} kind=gauge value=2 attrs.cache.name=customers
msg=metric name=cache.capacity unit={entry} kind=gauge value=0 attrs.cache.name=customers

If the service ran for more than a minute, you’ll also see an earlier block from a periodic report. The values are running totals, so read the last block.

Put together, those lines tell the story of the five requests:

  • Five requests, one failure. operation.duration counts 4 that worked and 1 labelled order_not_found, all finishing within 5 milliseconds.
  • Two hits, three misses. cache.operations shows the map already held the name twice: the second and third requests for order 1.
  • Three queries. db.client.operation.duration counts 2 queries that found an order and 1 that didn’t. Three queries for three misses… the database only saw what the cache couldn’t answer.
  • Two entries. cache.size is 2, for orders 1 and 2. Order 99 wasn’t stored, because the database had no customer to give it. cache.capacity is 0 because we never set one, which means unbounded.
  • A quiet pool. One idle connection, none in use, nothing waiting.

These are the OpenTelemetry names a collector would receive. The metric names reference lists them all, and Slog record fields explains each field on the line.

What we built

We started with a service that answered curl and said nothing else. It now reports:

  • how many requests it served, how many failed and why (by error kind, not message), and how long they took;
  • how many of those requests reached the database, and how long the queries took;
  • whether the connection pool is busy, and whether anything is waiting for it;
  • how often the cache can skip the database, and how big it has grown.

And the code that does the actual work barely changed. The handler still returns the responses it always did, errors still come back untouched, and the instrumentation sits around the work rather than inside it: one wrapper round the request, one round the query, and a hit or a miss where the map is checked.

Where to go from here

We used otelcore.Slog so the output could be read by eye, but the instrument code doesn’t depend on it. Give the helpers a meter from any provider and they behave the same way.

Built with Hugo · Theme Stack designed by Jimmy