Observability is the ability to understand a system's internal state from the data it emits externally — its logs, metrics, and traces — so you can answer new questions about its behaviour without shipping new code to ask them. That's the software-engineering sense of the word, borrowed from control theory and adapted for distributed systems. The one-line test: if a novel failure happens in production right now, can you explain why using data you already collect? If you'd have to add logging and redeploy first, the system isn't observable yet.
The term comes from control theory (Rudolf Kalman, 1960), where a system is observable if you can infer its internal state from its outputs. Software borrowed the word around 2016 as microservices made the old approach — SSH into the box and read the log — impossible. When a single user request fans out across thirty services, no engineer can hold the whole system in their head. Observability is the discipline of making that system explainable from the outside.
This guide is the monitoring sense of the word: telemetry for distributed systems. It covers how observability differs from monitoring (they are not synonyms), the three pillars and why they are necessary but not sufficient, the OpenTelemetry standardization shift, why high-cardinality data is the whole point, the cost problem that is quietly blowing up engineering budgets, and how observability underpins SLOs and modern reliability engineering.
Key takeaways
- Observability is about unknown-unknowns — debugging failures you never predicted, not just watching for ones you did.
- Monitoring is a subset of it — dashboards and alerts cover known failure modes; observability is the richer telemetry that answers new questions.
- The three pillars are necessary but not sufficient — logs, metrics, and traces only become observability when shared context lets you correlate them.
- OpenTelemetry ended the lock-in — instrument your code once, then switch backends without rewriting anything.
- Telemetry is a cost center — high-cardinality data can rival the spend it observes, so manage the bill the way FinOps manages cloud spend.
Observability vs monitoring: the real distinction
The two words get used interchangeably and it causes real confusion in planning meetings. The distinction that actually matters is about the questions you can ask.
Monitoring watches for conditions you defined in advance. You know CPU exhaustion is bad, so you alert when it crosses 90%. You know 500s are bad, so you graph the error rate. Monitoring answers known-unknowns: questions you already knew to ask, whose answers you didn't yet have. It is essential, and you should keep doing it.
Observability is what lets you interrogate a system about failures you never anticipated — unknown-unknowns. One customer, in one region, on one app version, sees checkout hang, but only when their cart holds more than fifty items. You never built a dashboard for that because you couldn't have predicted it. Observability is the property that lets you slice the live telemetry by region, version, and cart size until the pattern falls out — no code change, no redeploy.
Put crudely: monitoring tells you that something is wrong; observability helps you find out why. Monitoring is a subset of observability, not a competitor to it. A mature practice runs both — tight alerting on the known failure modes, and rich, queryable telemetry for everything else.
The three pillars: logs, metrics, and traces
Observability is conventionally described as three data types. Each answers a different question, and each has a different cost profile.
Logs are timestamped records of discrete events — "user 4821 failed login at 14:03:22." They are the highest-fidelity signal and the most expensive to store at volume. Structured logs (JSON, not free text) are the ones you can actually query; unstructured logs are just expensive text you grep in a panic.
Metrics are numeric measurements aggregated over time — request rate, error percentage, p99 latency, queue depth. They are cheap to store and fast to query because they are pre-aggregated, which is also their weakness: once you have averaged a value across a minute, the individual events are gone and you can't ask new questions of them.
Traces follow one request across every service it touches, recording how long each hop took. In a microservices architecture, a trace is the only artefact that shows the whole journey of a request and where the latency actually accumulated — which is almost never where the on-call engineer first guesses.
Now the honest part: the three pillars are necessary but not sufficient. A logging platform, a metrics database, and a tracing tool that don't talk to each other is not observability — it is three data silos and a human doing correlation by hand at 3 AM. The value only appears when the pillars share context: a trace ID stamped on every log line, exemplars linking a metric spike to the traces behind it, consistent resource attributes across all three. That is why practitioners increasingly talk less about three pillars and more about a single wide, structured event you can slice any way you like. The pillars are storage formats; observability is the correlation between them.
OpenTelemetry and the end of vendor lock-in
For a decade, instrumenting your code meant importing a specific vendor's agent. Switching vendors meant re-instrumenting everything, so nobody did — and the vendors priced accordingly, knowing you were stuck.
OpenTelemetry (OTel) changed the economics. It is an open, vendor-neutral standard — a CNCF project, by activity second only to Kubernetes — for generating and exporting logs, metrics, and traces. You instrument once against the OpenTelemetry APIs, run a collector that receives and processes the data, and export it to whatever backend you like: a managed vendor, an open-source stack, or several at once. Change your mind next year and you swap an exporter config, not your application code.
For engineering leaders the takeaway is strategic rather than technical: standardizing on OpenTelemetry is the single most effective way to avoid being locked into an observability vendor whose invoice you will eventually want to renegotiate. The cost is real — the collector is another distributed system you have to run and tune — but the leverage it buys is worth it for almost any team past a handful of services.
High cardinality is the whole point
Cardinality is the number of unique values a field can take. HTTP status code is low cardinality (a few dozen values). User ID is high cardinality (millions). The combination of user ID, region, device, and build number is astronomically high — and that combination is exactly what you need.
High-cardinality, high-dimensionality data is what makes unknown-unknowns debuggable, because the interesting failures hide in specific combinations of dimensions. The checkout bug that only affects one app version, in one region, for carts over fifty items is invisible unless your telemetry can be grouped by all of those fields at once. Strip the cardinality out to save money and you have thrown away precisely the signal that resolves incidents.
This is where traditional metrics systems fall down. Metrics stored as pre-aggregated time series explode combinatorially: every unique combination of label values becomes a separate series to store and index. Put a user ID in a Prometheus label and you can spawn millions of series and take the system down — a failure mode common enough to have its own name, "cardinality explosion." The modern approach keeps raw, wide events with their high-cardinality context intact and aggregates at query time instead of ingest time. That flexibility is powerful, and it is also — not coincidentally — the root of the cost problem.
The observability cost problem
Past a certain scale, almost every engineering org has the same uncomfortable realization: the bill for watching the system is a serious fraction of the bill for running it. Observability routinely lands at 10–30% of total infrastructure spend, and for some data-heavy teams the telemetry costs more than the workload it observes.
The mechanism is simple. Vendors price on data volume and cardinality, and distributed systems emit staggering amounts of both. Every new service, container, and label multiplies the data. Then teams over-collect by default: debug logs nobody ever queries, every trace retained at full fidelity for months, metrics on dimensions no dashboard uses. The data grows faster than the value, and the invoice grows with the data.
The fix is to treat telemetry as a cost center you actively manage — the same discipline FinOps brings to cloud spend. Concretely: tier your retention (hot for days, cheap or dropped after), sample high-volume traces with intent (keep the errors and the slow ones, sample the boring successes), drop log streams nobody queries, and put a named owner against the observability bill so it gets reviewed monthly rather than discovered during budget season. Teams that do this routinely cut spend 30–60% with no loss of the signal that actually gets used in an incident.
Observability, SLOs, and SRE
Observability is not an end in itself; it is the measurement layer that makes reliability engineering possible. You cannot manage what you cannot measure, and every core SRE practice depends on measurement.
A Service Level Objective (SLO) is a target for a service level indicator — say, "99.9% of checkout requests succeed under 800ms over a rolling 28-day window." You cannot compute that indicator without trustworthy telemetry, which makes observability a prerequisite for having SLOs at all. Better still, good observability lets you alert on error-budget burn rate — how fast you're consuming your allowed unreliability — instead of paging on raw symptoms like a CPU spike that may not affect a single user. Symptom-based alerting is the primary reason the pager fires for things that don't matter; we unpacked that failure mode in the alert fatigue trap.
The relationship runs both ways. Observability without SLOs is a pile of beautiful dashboards nobody opens until something is already on fire. SLOs without observability are aspirational numbers you can't actually measure. Together they are the core loop of modern reliability work, which is why our SRE consulting engagements almost always start by fixing the telemetry before touching anything else.
How to build an observability practice
You don't roll this out by buying a tool and switching it on. The order of operations is what separates a working practice from an expensive dashboard graveyard.
- Instrument the critical paths first. Start with the journeys that generate revenue or trust — checkout, login, the core API. Add structured logs, RED metrics (rate, errors, duration), and trace spans there before trying to cover the whole estate. Full coverage is a multi-year goal; the critical paths are a two-week one.
- Standardize on OpenTelemetry. Adopt the OTel SDKs and collector so telemetry is generated in a neutral format from day one. Retrofitting it later is expensive; starting with it is nearly free.
- Propagate context everywhere. Make a trace ID and a consistent set of resource attributes flow through every service, queue, and background job. Correlation across the pillars is the entire value proposition, and it only works if context is unbroken.
- Define SLOs and alert on burn rate. Turn the telemetry into indicators, set objectives, and page on burn rate rather than raw thresholds. This is the step that converts data into decisions.
- Control cost from the start. Set retention tiers, sample high-volume traces, and drop unqueried logs before the bill teaches you to. Review it monthly, not annually.
- Make it a habit, not a project. Put instrumentation in the definition of done, review dashboards in every incident retro, and prune dead signals quarterly. Observability decays the moment nobody owns it.
Common observability mistakes
- Buying a tool and calling it a strategy. A dashboard product doesn't make a system observable; instrumentation and correlation do. The tool is maybe 20% of the work.
- Collecting everything "just in case." Undirected data collection is how the bill explodes and how the real signal drowns. Instrument deliberately, for questions you expect to ask.
- Three disconnected pillars. Separate logging, metrics, and tracing tools with no shared trace ID force humans to correlate by hand under incident pressure — exactly when they are worst at it.
- Alerting on causes, not symptoms. Paging on CPU or memory produces noise; paging on user-facing SLO burn produces signal. Fix the alerting philosophy or the pager will train the team to ignore it.
- No cost owner. Observability spend with nobody accountable only ever grows. It needs the same ownership as any other significant line item.
Not sure whether your problem is missing telemetry, three tools that don't talk to each other, or a bill nobody is watching? InfraZen runs a free 30-minute observability review that ends in honest advice on the highest-leverage fix first — instrumentation, correlation, SLOs, or cost. Book the review.
Related: SRE Consulting services · What is SRE? · What is DevOps? · What is FinOps? · The alert fatigue trap · DevOps vs SRE vs Platform Engineering · SRE for SaaS