Observability: metrics, logs and traces, and what each one answers

Monitoring answers "is it working?". Observability answers "why is it not?".

The difference matters because the second question is the one you ask during an incident, and it requires data you needed to have collected beforehand.

The three pillars and their questions#

Metrics answer "what" and "how much".

A number aggregated over time: request rate, error rate, latency by percentile, resource usage. Cheap to store, because aggregation throws the detail away. Excellent for alerting and dashboards.

Bad for investigating one specific case: precisely because the detail was thrown away in the aggregation.

Logs answer "what happened in this case".

An event with context. Expensive in volume, because it aggregates nothing.

And here is the most practical piece of advice in this article: structured logs. JSON, with consistent fields, always carrying the correlation identifier. A free-text log is impossible to query exactly when you need it most: during the incident, when you want to filter by customer, by route and by time window all at once.

Traces answer "where the time went".

A trace follows one request across every service and shows the call tree with the duration of each span. It is the tool that answers "why did this request take two seconds" in thirty seconds instead of two hours.

If you have a distributed architecture and no tracing, you are investigating incidents with one hand tied behind your back.

1. Metricswhat and how muchA number aggregated over time. Cheap, because aggregation throws the detail away. Good for alerts and dashboards, bad for one specific case.
2. Logswhat happened hereAn event with context. Expensive in volume, because it aggregates nothing. Structured, or impossible to query during the incident.
3. Traceswhere the time wentThe call tree of one request, with the duration of each span. Answers in thirty seconds what would take two hours.

What ties the three together#

The correlation identifier: the trace id.

It is born at the edge, travels in a header through every internal call, and shows up in every log.

With it, you look at an anomalous metric, find a slow trace from that period, and jump straight to the logs of that specific request, across every service. Without it, each pillar is an island.

If your system does not propagate a trace id, that is the first thing to implement. It is cheap, it is mechanical, and it changes the game. OpenTelemetry solves it in a standardised way in most languages.

The four golden signals#

If you are only going to have four metrics per service, make them these, from Google's SRE book:

Latency. By percentile, and separating successful requests from failed ones. A fast error can mask bad latency in the average.

Traffic. Requests per second. Without it, you cannot tell whether the drop in errors was an improvement or just nobody using the system.

Errors. Rate, not count. And be careful about what counts as an error: a client 4xx and a server 5xx tell different stories.

Saturation. How full the most constrained resource is. Queueing is the most honest indicator of saturation, more than CPU usage.

The USE method, to go one level down#

When the four signals point at a resource problem, use Brendan Gregg's method. For each resource (CPU, memory, disk, network):

Utilisation: how much of the time it is busy.

Saturation: how much work is waiting in the queue.

Errors: a count of failures.

The insight is that high utilisation with low saturation is fine: it means you are using what you paid for. The problem is saturation. A disk at 90% utilisation with no queue is healthy; a disk at 60% with a growing queue is not.

Alert on symptoms, not on causes#

This is the most common mistake in alert configuration.

"Checkout error rate went above 1%" is a symptom. It matters, the user is feeling it, somebody should wake up.

"CPU on machine 3 is at 80%" is a possible cause. It may mean nothing. It may be exactly the expected behaviour under load.

Cause alerts create fatigue. And a team with alert fatigue ignores the alert that mattered. That turns your alerting system into expensive noise.

The rule: alert on what the user perceives. Use causes as context for investigating after the alert fires.

And tie it to the SLO: if you defined that 99.9% of requests answer in under 300ms, the alert should be about burning the error budget too fast, not about an isolated thirty second spike.

The "it is slow" routine#

Method, not hunches:

1. Is it CPU or is it waiting? Look at the process's CPU utilisation. If it is low and it is still slow, it is waiting on I/O, on a lock or in a queue.

2. If it is CPU, take a profile and look at the flamegraph. The width of each bar is the time spent in that function and everything it calls. You are looking for wide bars. Reading one takes thirty seconds once you understand the axis, and it is the most efficient tool there is for finding where the time goes.

3. If it is waiting, the trace tells you on what.

4. If it is a lock, measure contention time. If you add CPU and throughput does not rise, this is where you are.

5. If it is a queue, find which one: the load balancer's, the connection pool's, the executor's, the message broker's.

Do not skip steps and do not start by optimising. Intuition about performance is wrong with a humbling frequency.

Cardinality: the cost trap#

A metric with one label per user generates one time series per user. With a million users, a million series.

That blows up the cost and takes down the metrics system. The rule: metric labels are for low cardinality dimensions (route, status, region, version). User, order and request identifiers belong in logs and traces, never in metrics.

It is the most expensive mistake made in observability, and it only shows up on the invoice.

Where to start#

If you have little today, in order of payoff:

1. A trace id propagated end to end, showing up in every log.

2. Structured logs with consistent fields.

3. The four golden signals per service, with latency by percentile.

4. Distributed tracing on the critical routes.

5. Alerts on symptoms, tied to an SLO.

One concrete exercise for this week: implement trace id propagation on a single route. Trigger an error. Confirm you can follow that identifier from the edge log all the way to the database log.

If you can, you are already ahead of most.