Observability consulting: alerts that mean something.
Prometheus, Grafana and OpenTelemetry, wired to SLO-based alerting. We start from the pager, instrument what your SLOs need, and hand over dashboards, alert rules and runbooks your team owns.
Key takeaways
- Start from the pager, not the dashboards: every existing alert gets a verdict — page, ticket, dashboard or delete — before anything new is added.
- SLOs decide what pages. Multi-window burn-rate alerts on user-facing symptoms replace threshold alerts on causes.
- OpenTelemetry keeps the backend a choice: instrument once, then route to Prometheus, Grafana, a SaaS tool or all three.
- Telemetry is a cost line. Cardinality budgets, sampling and retention tiers are designed in, not bolted on after the bill.
What an observability consultant actually does
An observability consultant makes production systems explainable and alerts trustworthy: reading what you collect and what pages, defining SLOs for the journeys users care about, instrumenting services to measure them, and rebuilding alerting so a page means a user is hurting. It ships as pull requests your team reviews — alert rules, Collector config, dashboards and runbooks, all in Git.
Most observability projects run the other way round: install a stack, import dashboards, leave alerting for later. Later never comes. We invert the order — decide what deserves a human, then collect what that decision needs.
Plenty of monitoring, not enough answers.
Teams rarely come to us short of monitoring. They have too much of the wrong kind:
- Hundreds of alerts on causes — CPU, disk, pod restarts — and a rotation that has learned to dismiss them.
- Dashboards per tool and per incident, none saying whether users are hurting now.
- Metrics, logs and traces with no shared context, so incidents start with copying timestamps between tabs.
- A proprietary agent in every service, which turns the next backend decision into a rewrite.
- A telemetry bill growing faster than traffic, thanks to a high-cardinality label nobody remembers adding.
The noise has a price. In the survey behind our alert fatigue teardown, 44% of organisations had a 2025 outage caused by a suppressed or ignored alert. More dashboards do not fix that. Changing what is allowed to page does.
Six deliverables, all in your repos.
Observability audit and alert inventory
Read-only first. Every alert gets a verdict: page, ticket, dashboard or delete.
- Alert inventory with owner and purpose
- Coverage gaps against your user journeys
- Ranked risks and a written plan
SLOs and burn-rate alerting
SLIs measured at the edge, targets with product sign-off, and alerts that page on budget burn, not on CPU.
- Multi-window, multi-burn-rate rules as code
- Fast burn pages; slow burn opens a ticket
- An error-budget policy leadership signs
OpenTelemetry instrumentation
One vendor-neutral layer: SDKs in your services, context end to end, a Collector you control.
- Auto-instrumentation first, manual spans where it matters
- Collector as agent and gateway
- Tail sampling that keeps errors and slow requests
Prometheus that survives growth
Prometheus set up for next year's estate: recording rules, high availability and long-term storage.
- Recording rules for SLIs and heavy queries
- HA pairs with Alertmanager routing
- Thanos, Mimir or a managed Prometheus for retention
Grafana dashboards as code
Fewer dashboards, each answering one question: is this service healthy, and if not, why?
- A golden-signal dashboard per service
- Provisioned from Git, reviewed like code
- Metrics linked to traces and logs in Tempo and Loki
Telemetry cost control
The observability bill treated like cloud spend: attributed, budgeted and reviewed, self-hosted or SaaS.
- Cardinality budgets per service
- Drop rules for series nobody queries
- Retention tiers and custom-metric review
Read first, then change.
Week 1 is read-only. Everything after ships in pull requests your engineers review.
- 01
Read
Alerts, dashboards, incident history and on-call load, read without touching anything. You get the alert inventory and ranked risks first.
week 1: read-only - 02
Define
Map the journeys that pay the bills, choose SLIs at the edge, and set SLO targets with product sign-off.
in writing: SLO document - 03
Instrument
Wire OpenTelemetry and Prometheus to measure exactly those SLIs, then close the gaps traces reveal.
delivered in pull requests - 04
Rewrite alerts
Replace thresholds with burn-rate alerts, demote causes to dashboards, delete the rest. Every page that remains links to a runbook.
alert rules in code - 05
Pair and hand over
Work live incidents alongside your engineers, then watch them run one alone. The runbooks are the deliverable.
runbooks as deliverables - 06
Operate, if you want
Keep the pager with us on a managed SRE retainer, with 24×7 on-call and a 15-minute acknowledgement SLA.
optional: managed sre
How SLO-based alerting works
SLO-based alerting pages a human only when users are losing reliability faster than the SLO allows. Instead of “CPU above 80% on app-server-3”, the page reads “checkout will exhaust its 30-day error budget within hours at this rate”. That sentence carries its own justification.
The mechanism is the multi-window, multi-burn-rate alert from Google's SRE Workbook chapter on alerting on SLOs. A fast burn catches a sharp outage within minutes and pages; a slow burn catches a gradual leak and opens a ticket. A long and a short window per condition make the alert fire on sustained burn and clear soon after.
Cause alerts change jobs rather than vanish: CPU, disk and replication lag become dashboard panels, no longer waking anyone. That is the symptoms-over-causes rule from the SRE book's monitoring chapter, and on our SRE engagements teams typically go from hundreds of alerts to under 20 high-signal ones. The four golden signals say what to watch; SLOs say when to page.
Do you need OpenTelemetry if you already run Prometheus?
For infrastructure metrics, no: Prometheus exporters cover those well. In your own services, yes. OpenTelemetry is an instrumentation standard, Prometheus a metrics backend. One SDK emits metrics, traces and logs with shared context; the OpenTelemetry Collector routes them to Prometheus, Tempo and Loki, or to Datadog, New Relic or Honeycomb. With the Collector in the middle, switching backends is a config change, not a re-instrumentation project.
Two rules keep it affordable. Attributes follow the semantic conventions, so a service name means the same thing in every tool. Unbounded values — user IDs, request IDs, raw URLs — stay off metric labels, because each label combination is a new time series (the Prometheus naming guide is explicit). Tail sampling keeps every error and slow trace.
New to the vocabulary? What is observability? covers the pillars, high cardinality and the cost problem from first principles.
The backend follows the workload.
Self-hosted or SaaS, the choice comes with a written rationale. Weighing the two? Start with Prometheus vs Datadog.
Hire an observability consultant or build in-house?
Build in-house when someone on the team has run production observability before and alerting can be redesigned without a deadline. Bring in outside help when the pager is burning people out, when a renewal, audit or vendor bill forces a decision this quarter, or when nobody has set up SLO-based alerting before.
| Build in-house | Bring in a consultant | |
|---|---|---|
| Experience | An owner who has done it in production | Nobody has run SLO alerting yet |
| Timing | No deadline; improvements can land over quarters | A renewal, audit, migration or bill needs an answer now |
| The pager | Tolerable; alerts are acted on | Burning people out; alerts are routinely dismissed |
| What you keep | Knowledge with the people you retain | Rules, dashboards and runbooks in your repos |
When not to hire us: one service on a SaaS tool that pages rarely and costs what you expected does not need a project; the free 30-minute review is enough. The build-or-buy math is in managed DevOps vs in-house; the questions to ask any vendor, us included, are in how to choose an SRE consultancy.
Observability that moved the uptime number.
A representative engagement, anonymised at the client's request; the numbers are the ones we measured. Prometheus and Grafana, burn-rate alerts tied to four journey SLOs, runbooks for the top 15 incident classes. Read the uptime engagement →
Common questions.
What is observability consulting?
Observability consulting is outside engineering help to make production systems explainable and their alerts trustworthy: auditing what you collect and what pages, defining SLOs for the journeys users care about, instrumenting with OpenTelemetry, running Prometheus and Grafana or tuning your SaaS tool, and handing over alert rules, dashboards and runbooks. It is bought as a scoped project or as part of an SRE retainer, not as headcount.
What is SLO-based alerting?
SLO-based alerting pages people when a user-facing service is burning its error budget too fast, instead of when a resource crosses a threshold. A fast burn-rate window catches sharp outages and pages; a slow window catches gradual degradation and opens a ticket. In our experience it is the single largest reduction in pager noise available without touching the architecture.
Do we have to move off Datadog or New Relic?
No. We are vendor-agnostic, and SaaS is often right for a team without people to run a metrics platform. What changes is how it is used: instrumentation moves to OpenTelemetry so the backend stays a choice, alerts move to SLOs, and custom metrics and log ingest are reviewed so the bill tracks traffic. If self-hosting would genuinely save money, the numbers come first; our Prometheus vs Datadog comparison shows the math.
How do you keep observability costs under control?
By treating telemetry like any other cloud spend: attribute it, budget it and review it. In practice: cardinality budgets per service, no user or request IDs on metric labels, drop rules for series nobody queries, tail sampling for traces, and retention matched to how long each signal is actually used.
How long does an observability engagement take?
It depends on how many services and teams are in scope, and it is scoped in writing after the read-only first week. For reference, our 90-day SRE implementation puts SLOs and burn-rate alerting in weeks 3 to 6, and the representative SaaS engagement rolled out Prometheus, Grafana and burn-rate alerts in the same window.
What does observability consulting cost?
Market ranges, not our price list: senior-led boutiques run $8K to $30K per month on retainer, independents quote $40 to $100 per hour from India and $120 to $250 in the US and Western Europe, and global integrators bill $150 to $400 per hour blended. A scoped observability project is usually fixed-price. The full breakdown is on our DevOps consulting rates page.
$ infrazen book --review
Fewer pages. Better ones.
Book a free 30-minute observability review. Bring your alert list and last month's incidents; we'll tell you what to delete, what to keep and what to measure.