Before this:Services & systemd
Logging & health checks
Key takeaways A running service has to tell you it’s alive. Logs record what it’s doing — structured logs (fields, not just text) are searchable. A health check endpoint lets a monitor poll “are you OK?”; a bad answer triggers a restart or removal from rotation. Together they’re how you know a deployment is healthy without watching it.
Once a service runs unattended, you can’t watch it by staring at a terminal. Logging and health checks are how it reports its own state.
Logs: the running narrative
Logs are the diary of a running service — what it did, what went wrong, when. Under systemd, a service’s output is captured by the journal, which you read with:
journalctl -u gophertrunk -f # follow GopherTrunk's live logs
For a container, the equivalent is docker logs -f gophertrunk. Either way, the logs
are your first stop when something misbehaves.
Structured logs — events recorded as fields rather than free-form sentences — are much easier to work with as a system grows: you can filter for every error, every event on a given channel, every entry in a time window, instead of grepping prose. (The Linux CLI module covers reading logs in depth.)
Health checks: a heartbeat you can poll
A log tells you what happened; a health check answers “are you working right now?” The service exposes a small endpoint that returns OK when healthy:
$ curl http://127.0.0.1:8080/api/v1/health
{"status":"ok"}
GopherTrunk exposes exactly this at /api/v1/health, and its
docker-compose file wires Docker to poll it:
healthcheck:
test: ["CMD", "wget", "--quiet", "--spider", "http://127.0.0.1:8080/api/v1/health"]
interval: 30s
timeout: 5s
retries: 3
Every 30 seconds Docker checks the endpoint; three failures in a row mark the container unhealthy, and a restart policy or orchestrator can then act.
Liveness vs readiness
Two subtly different questions a health check can answer:
| Check | Asks | If it fails |
|---|---|---|
| Liveness | Is the process alive and not stuck? | restart it |
| Readiness | Is it ready to serve requests yet? | hold traffic until ready |
Small deployments often use a single health endpoint for both; larger systems separate them so a service that’s starting up isn’t killed for not being ready yet.
Metrics, briefly
Beyond a yes/no health check, many services also expose metrics — numeric gauges
and counters (requests handled, errors, memory) at an endpoint like /metrics for a
tool such as Prometheus to scrape. GopherTrunk labels its compose service for exactly
this. Metrics turn “is it up?” into “how is it doing?” — the subject of
monitoring & updates.
Quick check: what happens when a service repeatedly fails its health check?
Recap
- Logs are a service’s running narrative — read them via
journalctlordocker logs; structured logs are searchable. - A health check endpoint lets a monitor poll “are you OK?” — GopherTrunk exposes
/api/v1/health. - Liveness (alive?) and readiness (ready to serve?) are distinct questions.
- Metrics at
/metricsanswer “how is it doing?” beyond a simple up/down.
Next up: keeping credentials safe — secrets and configuration management.
Frequently asked questions
What is a health check endpoint?
It’s a small HTTP endpoint — often /health — that a service exposes to report whether it’s working. A monitoring tool or orchestrator polls it periodically; a healthy response means keep running, a failure or timeout means the service is stuck and should be restarted or taken out of rotation.
What are structured logs?
Structured logs record each event as machine-readable fields (like key/value pairs or JSON) rather than free-form text. That makes them searchable and filterable — you can pull every error for a given call or timeframe — instead of grepping unstructured lines. They’re far easier to work with once a system is running at scale.