BLUF: Do not approve an AIOps investment from a single “MTTR reduced by X%” number. Split every incident into time to detect, time to diagnose, time to mitigate or repair, and—where it differs—time to full recovery. Each interval has a different owner, data source and automation lever. Then compare like-for-like incident cohorts using distributions, not only arithmetic means. Otherwise faster alerting can make the aggregate look better while root-cause investigation and repair—the expensive stages—barely change.
MTTR is useful for service restoration, but weak for attribution
Google Cloud’s DORA-aligned implementation defines time to restore service as one of four software-delivery metrics and measures the interval around service disruption and restoration. Its worked example treats SLO violation as the start condition and return to compliance as the end condition. That is useful for customer-impact reporting. It does not identify which part of response improved.
Google Cloud’s MTTR guidance gives the familiar formula: total resolution time divided by incidents resolved. The same guidance explicitly says detection, diagnosis and repair all contribute, and maps monitoring, investigation and automation to different bottlenecks. A blended average removes exactly that structure.
Google SRE’s incident-metrics handbook makes a stronger statistical objection. Incident durations are positively skewed, incident volumes are often low and variance is high. In its simulations, a real 10% duration improvement was frequently invisible in MTTR, while apparent improvements also occurred when nothing changed. Its practical recommendation is to tailor measurement to the question and study the specific incident-response activity a change targets.
Define one timeline before comparing tools
Use event time, not ticket-edit time. A practical timeline is: t0 = first verified user or service impact; t1 = detection by the operating system or responder; t2 = a diagnosis accepted strongly enough to choose a response; t3 = mitigation or repair that removes material impact; t4 = full recovery, including degraded replicas, backlog and cleanup.
Detection = t1−t0. Diagnosis = t2−t1. Repair/mitigation = t3−t2. Recovery tail = t4−t3. Customer-impact duration = t3−t0 when mitigation restores the SLO; full-recovery duration = t4−t0. State which definition you call MTTR. “Reported” is not a stable timestamp unless the reporting channel is fixed.
The evidence path should be: telemetry and SLO events → immutable incident event stream → normalized lifecycle events → stage-duration calculator → cohort store → percentile and paired-outcome analysis → ROI dashboard. Preserve source_event_id, incident_id, service, severity, failure mode, deployment/change ID, event_time, ingestion_time, actor, automation version, confidence and override reason. The prerequisite is passive, time-accurate observability evidence, not another dashboard.
Required visual: the aggregate bar hides the result
Illustrative stacked bars, same incident cohort and severity mix: BEFORE AIOps, total 60 min = DETECT 10 + DIAGNOSE 35 + REPAIR 15. AFTER AIOps, total 48 min = DETECT 4 + DIAGNOSE 31 + REPAIR 13. The headline says “20% lower MTTR.” The decomposition says detection improved by 60%, diagnosis by about 11%, and repair by about 13%; diagnosis still consumes 31 of 48 minutes. This is an example, not a benchmark.
| BEFORE | Detect █████ 10 | Diagnose █████████████████ 35 | Repair ███████ 15 | 60 min |
|---|---|---|---|---|
| AFTER | Detect ██ 4 | Diagnose ███████████████ 31 | Repair ██████ 13 | 48 min |
Procurement implication: pay for the detection result only if detection was the business bottleneck. If the purchase case promised automated root-cause analysis or remediation, the example has not established that value.
Decision table: metric, lever, and proof
Stage: detect. Timestamp pair: first impact → first actionable detection. Primary lever: SLO-based alerting, anomaly detection, deduplication, routing. Proof: lower p50/p90 detection delay at controlled false-positive and missed-incident rates. Failure mode: the tool starts the clock at its own alert, erasing pre-alert customer impact.
Stage: diagnose. Timestamp pair: accepted detection → documented working diagnosis that selects an action. Primary lever: correlation, topology context, causal traversal, log/trace enrichment and hypothesis support. Proof: shorter p50/p90 diagnosis time plus first-hypothesis accuracy and fewer hypothesis reversals. Failure mode: “AI root cause” is counted when suggested, not when evidence validates it. For architecture choices, see causal AI versus correlation in AIOps.
Stage: repair or mitigate. Timestamp pair: accepted diagnosis → SLO restored. Primary lever: runbook execution, rollback, traffic shift, feature isolation or approved operator action. Proof: shorter execution time, automation success rate, rollback rate and recurrence within a fixed window. Failure mode: automation is fast but applies the wrong action or causes a second incident. The control pattern is detailed in approval-gated incident automation.
Stage: recovery tail. Timestamp pair: mitigation → full service and backlog recovery. Lever: capacity restoration, data reconciliation and cleanup orchestration. Proof: tail duration and residual customer impact. Failure mode: the MTTR clock stops at partial mitigation while queues, replicas or customers remain degraded.
How to instrument the lifecycle without creating fiction
Generate t0 from an SLI/SLO or independently verified impact event where possible. Record both system detection and human acknowledgement; they answer different questions. Require responders to mark diagnosis only when they can name the suspected failed component or mechanism, cite evidence and select an action. Log runbook start, every side effect, approval, outcome and rollback automatically.
Human labels remain imperfect. Use controlled vocabulary, but retain the raw timeline and post-incident corrections. A diagnosis can change; store revisions rather than overwriting the first hypothesis. Separate machine-generated suggestions from responder acceptance. If confidence is low, mark the stage boundary uncertain and exclude it from causal ROI claims rather than manufacturing precision.
Join tool exposure at incident level: AIOps version, enabled features, topology coverage, runbook availability and whether responders actually used the recommendation. “After deployment” is not the same as “treated.” Without exposure data, attribution collapses into a calendar comparison vulnerable to staffing, architecture and incident-mix changes.
Evaluation design for an AIOps business case
1. Freeze metric definitions and clocks before the pilot. 2. Stratify by service, severity and failure mode. 3. Report counts, p50, p75 and p90 for every stage; show the full distribution when samples are small. 4. Compare matched or randomized treatment where operationally possible. 5. Replay historical incidents for deterministic detection and diagnosis tests, then validate live workflow effects separately. 6. Track customer impact, false positives, unsafe actions and recurrence as guardrails.
Bootstrap confidence intervals or a pre-specified non-parametric test can quantify uncertainty, but no statistical technique repairs incomparable cohorts. One rare multi-hour incident can dominate a mean. Do not delete it merely to improve the result; report robust percentiles and a sensitivity analysis with and without declared extreme events.
Translate stage change into value only after measurement: avoided customer-impact minutes × an agreed cost range; responder minutes saved × loaded labor cost; and avoided SLA or operational losses where finance can substantiate them. Subtract licenses, integration, telemetry storage, model calls, runbook maintenance, false-positive handling and governance effort. Do not monetize faster detection twice through both downtime and labor.
Failure modes that invalidate the ROI claim
Clock substitution: vendor alert time replaces first impact. Cohort drift: the “after” period contains easier or lower-severity incidents. Survivorship bias: unresolved incidents are omitted because the formula counts only resolved cases. Partial restoration: mitigation is labelled recovery. Parallel work: detect, diagnose and repair overlap, but the model forces a clean sequence without rules. Tool exposure leakage: incidents where the tool was disabled are counted as treated.
Automation-quality blindness: execution gets faster while rollback and recurrence rise. Topology-coverage bias: only well-modelled services enter the pilot. Mean-only reporting: a small number of long incidents swings the result. Human process change: a new on-call rota or runbook library receives no credit. Goodhart pressure: teams close tickets early to hit MTTR targets. Each failure needs an audit query and an owner before procurement review.
Trade-offs and limitations
A decomposed model is more useful, not perfectly causal. Incident stages overlap; mitigation can start before root cause is known; distributed failures may have several causes; and “diagnosis accepted” includes human judgment. Instrumentation adds responder burden unless events are captured from monitoring, chatops, runbooks and deployment systems automatically.
Small samples remain noisy even after decomposition. Stage metrics can invite local optimization: aggressive alerting lowers detection delay but increases false positives; fast diagnosis can become overconfident; automatic repair can increase change risk. Pair every speed metric with quality and safety guardrails. Reliability itself is better represented by user-facing SLIs and SLOs than by incident-response speed alone.
Implementation checklist
— Agree t0–t4 and distinguish mitigation from recovery. — Keep event time and ingestion time. — Preserve unresolved and censored incidents. — Stratify by service, severity and failure mode. — Record tool exposure, version, topology coverage and recommendation use. — Report stage p50/p90, counts and confidence intervals. — Add false-positive, diagnosis-accuracy, unsafe-action, rollback and recurrence guardrails. — Reconcile stage sums with customer-impact duration. — Audit boundary edits. — Require the vendor to reproduce every claimed percentage from exportable incident-level data.
Practical next step
Take the last 20–30 material incidents and reconstruct t0–t4 from immutable evidence. The immediate deliverable is a coverage report: which timestamps exist, which are trustworthy, and where diagnosis or repair events remain subjective. Only then set a baseline and pilot target. I help platform and operations teams turn that timeline into an auditable AIOps evaluation and a business case that finance, SRE and engineering can challenge.
Primary and authoritative sources
Google Cloud: DORA-aligned DevOps performance measurement — time to restore service, SLO/SLI start and clearing events.
Google SRE: Incident Metrics in SRE — skewed durations, Monte Carlo evidence, limits of MTTx and narrower activity metrics.
Google Cloud: How to Reduce MTTR — formula and distinct detection, diagnosis and repair levers.
