Before this:Logging & health checks
Observability — metrics, logs & traces
Key takeaways
Observability is being able to understand a running system from the outside. It rests on
three pillars: metrics (cheap numbers over time — “is it up?”), logs
(timestamped events — “what happened?”), and traces (one request’s path across
services — “why is it slow?”). GopherTrunk exposes a /metrics endpoint that
Prometheus scrapes and Grafana graphs.
Logging & health checks gave you a service that can say it’s healthy. Observability is the wider skill of understanding it — answering questions you didn’t script in advance. Three kinds of signal do that, and each answers a different question.
The three pillars
Metrics: is it up, and how much?
Metrics are cheap numeric measurements sampled over time — request rate, error count,
response latency, memory in use. They’re compact enough to keep for months, which makes
them perfect for dashboards, trends, and alerts.
GopherTrunk exposes them in Prometheus format at /metrics — its compose file even
labels the container for a Prometheus sidecar to find:
labels:
- "prometheus.scrape=true"
- "prometheus.port=8080"
- "prometheus.path=/metrics"
Prometheus periodically scrapes that endpoint and stores the numbers as time-series; Grafana graphs them into dashboards:
# a Prometheus scrape target
scrape_configs:
- job_name: gophertrunk
static_configs:
- targets: ["127.0.0.1:8080"]
metrics_path: /metrics
A metric like http_requests_total climbing while http_request_errors_total stays flat
tells you at a glance the service is healthy under load — no log-reading required.
Logs: what exactly happened?
Metrics tell you that errors rose; logs tell you what they were. A log line is a timestamped record of a discrete event, ideally structured so you can filter it:
{"time":"2026-07-23T14:02:11Z","level":"error","msg":"decode failed","tg":52198,"freq":851012500}
When a metric spikes, logs are where you go to read the story of a specific failure — the talkgroup, the frequency, the error. Ship them somewhere searchable (the journal, or a log aggregator) so you can query across time rather than tailing one file.
Traces: where did the time go?
In a system of several services, a single user request hops between them, and “why is it slow?” means “which hop ate the time?” A trace follows one request end to end, recording how long each step took as nested spans. For a single-service app like a lone GopherTrunk instance, traces matter little — there’s one process. They earn their keep once a request fans out across services, which is exactly the world orchestration and scaling create.
Pick the pillar for the question
| The question | The pillar |
|---|---|
| Is the service up? Is error rate rising? | Metrics |
| What was that specific failure? | Logs |
| Which service made this request slow? | Traces |
| Should I get paged right now? | Metrics → alerts |
You don’t need all three from day one — metrics plus good logs cover most single-service deployments, and GopherTrunk gives you both out of the box. Add traces when you have multiple services and “which one is slow?” becomes a real question.
Quick check: which pillar best answers "is the service up and how much load is it under?"
Recap
- Observability rests on three pillars: metrics, logs, and traces.
- Metrics are cheap numbers over time — “is it up?” — and drive dashboards and alerts;
GopherTrunk exposes
/metricsfor Prometheus, graphed by Grafana. - Logs are timestamped events — “what happened?” — best kept structured and searchable.
- Traces follow one request across services — “why is it slow?” — and matter most in multi-service systems.
Next up: turning metrics into pages that matter — alerting and on-call.
Frequently asked questions
What are the three pillars of observability?
Metrics, logs, and traces. Metrics are cheap numeric time-series that answer ‘is it up and how much?’ — request rate, error rate, memory. Logs are timestamped event records that answer ‘what exactly happened?’ Traces follow one request across services to answer ‘where did the time go?’ Together they let you understand a system’s behaviour from the outside without attaching a debugger.
What is the difference between monitoring and observability?
Monitoring watches for known problems — you decide in advance what to measure and alert on. Observability is the broader property of being able to ask new questions of a running system after the fact, including ones you didn’t anticipate. Good metrics, logs, and traces give you observability; dashboards and alerts built on them are monitoring.