The four golden signals of observability:
latency, traffic, errors, saturation.

A stock Kubernetes cluster with a normal-sized Prometheus setup exposes tens of thousands of metric series before you've instrumented a single line of your own code. That abundance is the problem. When the checkout flow slows down at 2 AM, nobody is paging through ten thousand series; the on-call engineer needs four or five charts that say, quickly and honestly, whether users are in pain and where the pain is coming from.

Google's answer to that problem has held up for a decade. The four golden signals of observability are latency, traffic, errors, and saturation — how long requests take, how much demand the system is serving, what fraction of requests fail, and how close the system is to its capacity ceiling. The Site Reliability Engineering book puts it bluntly: “If you can only measure four metrics of your user-facing system, focus on these four.”

Ten years on, the four signals are the default template in most monitoring products and the first dashboard on most walls. They're also widely misapplied, because the industry compressed a careful ten-page chapter into a bumper sticker. This post unpacks what each signal actually means (including the subtleties that bite), where the RED and USE methods fit, how the signals turn into alerts that don't burn out your on-call rotation, and what instrumenting them looks like on Kubernetes with Prometheus and OpenTelemetry.

1. Where the golden signals come from, and why they won

The four signals appear in the “Monitoring Distributed Systems” chapter of Google's 2016 SRE book, written by Rob Ewaschuk. The chapter is about a harder question than metrics: out of everything a distributed system can tell you, what deserves human attention? Its answer was a deliberately small, user-centred set — and that framing is exactly why it beat the “monitor everything, alert on most of it” culture that preceded it.

The four signals won because they are symptoms, not causes. High CPU is a cause; it may or may not matter. Slow checkouts, failing logins, and a queue that's filling faster than it drains are symptoms — users are feeling them right now. The signals also scale in both directions: they describe a whole platform, a single microservice, or one endpoint equally well, and they're vendor-neutral, which is why every generation of tooling since — Prometheus, Datadog, Grafana, OpenTelemetry — has re-shipped them as a default.

2. The four signals, properly defined

Here's the one-screen version, followed by the details that separate a useful implementation from a decorative one.

Latency: percentiles or nothing

Latency is the time it takes to service a request — and it's the signal teams most often measure wrong, in two specific ways the SRE book calls out directly.

First: separate the latency of successful and failed requests. An HTTP 500 served in 15 milliseconds by a tripped circuit breaker improves your blended average while your users stare at an error page. The inverse is nastier: slow errors, like a 30-second timeout before the failure, hold connections open and drag the whole service down with them. One latency series per outcome, or the signal lies.

Second: averages hide the tail, and the tail is where your best customers live. A 100ms average is compatible with a p99 of 4 seconds — and the requests in that tail are disproportionately your heaviest users, because they make the most calls. Instrument latency as a histogram, watch p50, p95, and p99 as separate lines, and set targets on the percentile that matches your SLO, not on the mean.

Traffic: the denominator, and the silence detector

Traffic measures demand, in whatever unit your system actually serves: requests per second for an API, transactions per second for a payment switch, messages consumed per second for a queue worker, concurrent streams for a media service. Pick the unit that maps to business demand, because that's the number everyone from on-call to the CFO can reason about.

Traffic earns its golden status two ways. It's the denominator that gives every other signal meaning — 40 errors per minute is an incident at 100 req/s and a rounding error at 100,000. And it detects a class of outage nothing else catches: when traffic drops to zero, your service metrics all look perfect while nobody can reach you. DNS misconfigurations, load-balancer changes, a broken mobile release, an expired certificate — the first symptom of all of them is silence.

Errors: defined by the user, not the process

The error rate is the fraction of requests that fail — but “fail” needs a user-centred definition, and the book's own taxonomy is worth keeping: errors can be explicit (an HTTP 500, a refused connection), implicit (an HTTP 200 whose body is wrong, empty, or stale), or a policy failure (a correct answer that arrived slower than your SLO promises — if you committed to one second, a three-second success is an error).

Where you measure matters as much as what you count. The load balancer sees errors your application logs never will — requests that died before reaching a healthy backend — and your client-side telemetry sees failures the load balancer never will, like a CDN serving a broken bundle. Measure at the edge as your source of truth, and per-service beneath it for attribution.

Saturation: how full, and how long until full

Saturation asks how full your service is — utilization of the resource that constrains it. For a CPU-bound service that's CPU; for a database it might be disk I/O, connections, or replication capacity; for a worker fleet it's queue depth versus drain rate. If you don't know which resource constrains a service, that's the first finding of your observability work, not a detail to skip.

Two properties make saturation different from the other three. It leads rather than lags: queueing effects mean latency starts degrading well before utilization touches 100%, so a saturation trend is your early warning while latency is still green. And it's the natural home of predictive alerting — the SRE book's own example is “it looks like your database will fill its hard drive in 4 hours.” That page at 4 PM is a calm ticket; the same discovery at 3 AM is an outage.

3. Golden signals vs RED vs USE: same idea, different layers

The four signals share shelf space with two sibling acronyms, and choosing between them confuses more teams than it should. The short version: they're not competing standards, they're the same idea applied at different layers of the stack.

RED is essentially the golden signals with saturation dropped and the names tightened — deliberately, because Tom Wilkie wanted one identical dashboard for every request-driven microservice, and per-service saturation doesn't generalise as cleanly. USE comes from the other direction: Brendan Gregg's checklist for interrogating resources — for every resource, check utilization, saturation, and errors — which is exactly what you want for a node, a disk array, or a connection pool, and exactly wrong as a description of user experience.

In practice, a Kubernetes shop ends up using all three without ceremony: RED-style dashboards per service, USE-style dashboards per node and per pool, and the golden signals as the top-level view of each user-facing system. If you're choosing where to start: start golden, because it's the only one of the three that forces you to look at saturation and the request path on the same page.

4. Signals are what to watch. SLOs are when to page.

Here's where most golden-signals rollouts go wrong. A team instruments all four signals for forty services, sets static thresholds on each — and has just created 160 ways to be woken up. Four signals × forty services is a dashboard strategy, not an alerting strategy. We've written before about where that road ends: 44% of organizations in one 2025 survey had an outage caused directly by an alert someone had learned to ignore.

The pattern that works treats the signals as raw material, not as alarms:

• Latency and errors become SLOs. Define them as good-event ratios (“99.9% of requests succeed in under 800ms over 30 days”) and page on error-budget burn rate, with a fast window for cliff-edges and a slow window for slow bleeds. Two alerts per SLO, every page actionable.
• Saturation becomes tickets and forecasts. Sustained headroom erosion and “disk full in N hours” predictions belong in working hours, not at 3 AM — they're urgent, but they're rarely right-now urgent.
• Traffic becomes context and anomaly detection. A traffic cliff pages (that's your silence detector); everything else about traffic is capacity planning.

If you're setting SLO targets for the first time, our free uptime & SLA calculator converts a target like 99.9% into its real-world downtime budget, which makes the conversation with product concrete instead of theological.

5. Instrumenting the signals on Kubernetes

On a modern stack, most of this is assembly rather than invention. Instrument services with OpenTelemetry or a Prometheus client library, and the four signals fall out of two instruments: a request-duration histogram and a request counter with an outcome label.

• Latency — a percentile from the duration histogram: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))). One caveat that ruins more dashboards than any other: default histogram buckets rarely bracket your SLO threshold. If your target is 800ms and your buckets jump from 500ms to 1s, your “p99” is interpolation fiction. Define buckets around the thresholds you actually care about.
• Traffic — the counter's rate: sum(rate(http_requests_total[5m])), sliced by route or consumer as needed.
• Errors — the failing fraction of the same counter: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) — plus whatever your “implicit error” definition adds, like a validation-failure counter for 200-but-wrong responses.
• Saturation — usage against limits from cAdvisor and kube-state-metrics: sum by (pod) (rate(container_cpu_usage_seconds_total[5m])) divided by kube_pod_container_resource_limits{resource="cpu"}, with container_memory_working_set_bytes against memory limits, node-exporter for node-level pressure, and queue depth for anything asynchronous.

Two shortcuts worth knowing. A service mesh — Istio or Linkerd — emits per-service rate, error, and duration metrics for every workload it fronts, without touching application code; that's three of the four signals for free, and honest ones, because they're measured from outside the process. And OpenTelemetry's HTTP semantic conventions standardise the request-duration histogram (http.server.request.duration), so instrumentation you add today stays portable across whatever backend you're on — a big part of why we push clients toward OTel in our Prometheus vs Datadog comparison.

6. Five mistakes we keep seeing

From observability engagements over the past few years, the recurring failure modes are remarkably consistent:

• 1. Averaging latency. One blended mean, successes and failures mixed together. It hides the tail, and it moves in the wrong direction during some failures (fast errors pull it down).
• 2. Alerting on raw signals. Static thresholds per signal per service is how you build the 160-alert catalogue nobody reads. Page on SLO burn; dashboard the rest.
• 3. Measuring only at the edge. A load-balancer-level view can't tell you which of twelve services burned the latency budget. Golden signals are fractal: apply them per service and per critical dependency, not just at the front door.
• 4. Treating saturation as someone else's problem. Teams instrument the three request-side signals (RED culture makes this easy) and skip the one signal that would have warned them before the incident. If your postmortems keep saying “we ran out of X,” this is why.
• 5. Stopping at four charts. The signals tell you that something is wrong. Finding why takes high-cardinality attributes, traces, and logs — the difference between monitoring and observability proper. The four signals are the floor of a practice, not the practice.

The takeaway

The golden signals have survived a decade of tooling churn because they're the smallest set of questions that still describes both halves of a production system: what users are experiencing (latency, traffic, errors) and how much runway the system has left (saturation). Every monitoring fad since has been a re-slicing of the same four questions.

The implementation that works is unglamorous: instrument all four per service, page on latency and error SLO burn rates, trend saturation with predictive tickets, and treat traffic as the denominator that keeps everything else honest. Four signals, two pages per SLO, and an on-call rotation that trusts what wakes it up.

Got four dashboards and forty alerts, but no confidence in either? InfraZen runs observability reviews as part of our SRE practice: we map your golden signals per service, define latency and error SLOs with burn-rate paging, and hand you the saturation forecasts that turn 3 AM outages into 4 PM tickets. Book a free 30-minute observability review and we'll tell you honestly which signal you're missing.

Frequently asked questions

What are the four golden signals of observability?

The four golden signals are latency, traffic, errors, and saturation. They come from Google's Site Reliability Engineering book, which advises: if you can only measure four metrics of your user-facing system, focus on these four. Latency measures how long requests take, traffic measures demand on the system, errors measure the rate of failed requests, and saturation measures how close the system is to its capacity ceiling.

Who created the four golden signals?

They were popularised by Google's Site Reliability Engineering book (2016), in the Monitoring Distributed Systems chapter written by Rob Ewaschuk. The chapter distilled how Google's SRE teams monitored large distributed systems, and the four-signal shorthand spread from there to become the default starting point for service monitoring across the industry.

What is the difference between the golden signals, the RED method, and the USE method?

RED (rate, errors, duration), from Tom Wilkie, describes request-driven services and is roughly the golden signals minus saturation. USE (utilization, saturation, errors), from Brendan Gregg, describes physical resources like CPU, memory, disks, and network. The golden signals span both layers: latency, traffic, and errors describe the request path, while saturation describes the resources underneath it.

Are the four golden signals enough for observability?

No. The golden signals are the minimum viable dashboard, not complete observability. They tell you that something is wrong; they rarely tell you why. Full observability adds high-cardinality attributes, distributed traces, and logs so you can debug failures you never predicted. Treat the four signals as the floor of your practice and SLO-based alerting as the layer that decides when a human gets paged.

How do you monitor the golden signals in Kubernetes?

Instrument services with OpenTelemetry or Prometheus client libraries for latency, traffic, and error metrics, and take saturation from cAdvisor, node-exporter, and kube-state-metrics: container CPU and memory usage against limits, plus queue depth for asynchronous work. A service mesh like Istio or Linkerd can emit per-service rate, error, and duration metrics without any code changes.

Related: What is observability? (definition guide) · The alert fatigue trap · Prometheus vs Datadog · Uptime & SLA calculator · SRE consulting