This topic ties together everything from the rest of the module into a single, realistic incident from first alert to postmortem draft. It's written as a runbook you can adapt directly — the specific service names and numbers are illustrative, but the sequence of tool calls, the prompts, and the verification checkpoints are the actual shape of a real investigation using Datadog MCP with any of the agents covered in this module (Claude Code, OpenCode, Gemini CLI, Cursor — the workflow is agent-agnostic; only the client-specific setup differs).
The scenario: it's 03:12 UTC, a monitor named checkout-error-rate-high has fired, and you're the on-call engineer waking up to a Slack alert with no other context yet.
Workflow Overview: From Monitor Alert to Root Cause and Postmortem
The workflow has four phases, and the point of using an MCP-connected agent is that phases 1-3 — which traditionally mean opening four or five different Datadog UI tabs, manually copying timestamps and trace IDs between them — collapse into a single conversation where each tool call's output feeds the next prompt.
- Pull context. What fired, what's the monitor actually measuring, which services and hosts are affected, what do the error logs say.
- Correlate. Cross-reference traces, deploy events, and infrastructure metrics against the same time window to find the causal chain, not just correlated symptoms.
- Confirm and mitigate. Verify the root cause with a targeted trace or log query before acting, then take (or hand off) the mitigation.
- Document. Generate a timeline and action-item draft while the details are still fresh and the tool outputs are still in context — waiting until the next day loses precision.
A structural point worth internalizing: at no phase should you let the agent skip straight to "here's the root cause" without showing you the data behind it. The value of MCP here is faster data retrieval and correlation, not replacing your judgment about what the data means. Every prompt below asks for evidence, not just a conclusion.
Tips
- Treat the agent as a very fast research assistant that fetches and cross-references data, not as a replacement for your own causal judgment — always ask for the evidence alongside any claim.
- Keep the entire investigation in a single conversation/session so later prompts (timeline generation, action items) have full context from earlier tool calls.
- Note the exact monitor name from the alert immediately — you'll need it verbatim for your first prompt, and guessing at a similar-sounding monitor name is a common early mistake.
Step 1: Pulling Monitor Context, Affected Services, and Error Logs
First prompt, immediately after opening your agent session:
The monitor "checkout-error-rate-high" just fired. Show me its query
definition, current status, and the services/tags it's scoped to.
This grounds the investigation in exactly what triggered — not a guess about what "checkout" problems might mean. A typical response confirms the monitor's underlying query, something like:
sum(last_5m):sum:trace.express.request.errors{service:checkout-api,env:production}.as_count()
> 50
along with current status Alert, and the fact it's scoped to checkout-api in production specifically — already ruling out staging noise.
Second prompt — pull the actual error content, scoped tightly using what you just learned:
Search logs for service:checkout-api status:error env:production for
the last 20 minutes, group by @error.message, and show top 5 with counts.
Say this surfaces:
1. "Connection timeout to payment-gateway" — 340 occurrences
2. "Unhandled promise rejection in processPayment" — 12 occurrences
3. "Redis connection refused" — 4 occurrences
The dominant error (340 vs the others) tells you where to focus first — payment gateway connectivity, not the smaller Redis or promise-rejection noise, which are very plausibly downstream symptoms of the same root issue rather than independent problems.
Third prompt — check if this is isolated or part of something wider:
Are there other monitors currently in Alert or Warn state that might
be related — anything scoped to payment-gateway or shared infrastructure?
If a payment-gateway-latency monitor is also alerting, that's strong confirmation you're looking at one incident with a single upstream cause, not two coincidental problems — an important distinction for both your mitigation plan and your postmortem's root-cause section.
Tips
- Always pull the monitor's actual query definition first — it tells you the precise scope (service, env, threshold) rather than you guessing from the monitor's name alone.
- Rank error messages by count before investigating each one — the highest-count error is usually (not always) the primary issue, with smaller counts being downstream symptoms.
- Check for other currently-alerting monitors early; a cluster of related alerts firing together is a strong signal you're dealing with one upstream root cause, not several unrelated ones.
Step 2: Correlating Traces, Deploy Markers, and Infrastructure Metrics
With payment-gateway connectivity as the lead, the next phase is confirming the actual failure point and ruling in or out a deploy as the trigger.
Pull 5 recent traces for service:checkout-api resource:"POST /checkout/complete"
from the last 20 minutes where the trace includes an error, and show me
the span for the payment-gateway call specifically — its duration and
any error tags.
A useful result here isolates the exact span: say a http.request span to payment-gateway consistently shows either a timeout at exactly 5000ms (pointing at a client-side timeout configuration being hit) or a connection-refused error (pointing at the gateway being fully down, not just slow). These are different root causes requiring different responses, and the trace data — not the log message alone — is what distinguishes them.
Were there any deploys to checkout-api or payment-gateway in the last
2 hours?
Two branches from here:
If a deploy correlates in time — say payment-gateway deployed 25 minutes ago, right before errors started — that's your leading hypothesis, and rollback is the fastest mitigation to consider. Confirm before rolling back:
Get avg:trace.express.request.duration{service:payment-gateway} and
error rate for the last 3 hours, and mark the deploy timestamp on
the timeline — does the change align cleanly with the deploy, or did
degradation start before it?
If no deploy correlates, pivot to infrastructure:
Get avg:system.cpu.user, avg:system.mem.used, and
avg:aws.elb.healthy_host_count for the payment-gateway host group over
the last 2 hours.
Suppose this reveals healthy host count dropped from 6 to 2 around the time errors began — that reframes the incident entirely, from "code regression" to "infrastructure capacity event" (autoscaling failure, host crashes, a bad AZ), which points at a completely different team and a completely different fix.
Either way, the discipline is the same: don't accept "it's probably the deploy" or "it's probably infra" as a conclusion until a specific metric or trace confirms the timing lines up precisely, not just approximately.
Tips
- Let the specific error type in the trace span (timeout value vs connection-refused) distinguish between "the dependency is slow" and "the dependency is down" — these look similar in a log message but very different in trace data.
- Always ask whether degradation started before or after a candidate deploy timestamp — "correlates" needs the ordering to be right, not just proximity.
- If no deploy correlates, pivot to infrastructure metrics (host count, CPU, memory) before assuming code is at fault — a capacity or infra event can produce identical symptoms to a bad deploy.
Step 3: Generating the Incident Summary and Follow-Up Action Items
Once you've confirmed a root cause and applied or initiated a mitigation (rollback, scale-up, failover — whatever the specific case calls for), close the loop with documentation while the tool outputs are still live in the session.
Generate an incident timeline table using every timestamp we've
established in this conversation: first anomaly, monitor fire time,
root cause confirmation, and mitigation time. Cite the specific query
or trace ID behind each row.
Expect something like:
| Time (UTC) | Event | Source |
|---|---|---|
| 02:58 | payment-gateway healthy host count drops 6→2 | avg:aws.elb.healthy_host_count |
| 03:02 | checkout-api error rate begins climbing | log count, status:error |
| 03:12 | checkout-error-rate-high monitor fires | monitor status API |
| 03:19 | Root cause confirmed: infra capacity, not deploy | trace + deploy marker check |
| 03:27 | Manual scale-up triggered for payment-gateway ASG | (manual action, not queried) |
| 03:41 | Error rate returns to baseline | log count, status:error |
Follow with a direct request for action items — and push for specificity, since a vague action item ("improve monitoring") is worthless in a postmortem:
Based on this incident, draft 3-5 concrete follow-up action items.
For each, specify: what breaks again without this fix, who likely
owns it, and whether it's a monitoring gap, a code fix, or an
infrastructure change.
A good output distinguishes action types clearly rather than producing generic advice:
1. [Infrastructure] Add autoscaling alarm on payment-gateway healthy-host-count
dropping below 4 — would have caught this 14 minutes before the
customer-facing error monitor fired. Owner: platform team.
2. [Monitoring] Add a monitor directly on payment-gateway health,
separate from checkout-api's downstream error rate — we currently
only see this class of issue after it cascades to checkout.
Owner: on-call/SRE.
3. [Code] Review checkout-api's timeout/retry configuration for
payment-gateway calls — current 5s timeout with no retry may be
too aggressive or too passive; needs a deliberate decision either way.
Owner: checkout team.
Before this goes into your actual incident tracker or postmortem doc, verify every row against the source data yourself — particularly the "would have caught this N minutes before" style claims, which are a specific, checkable number the agent computed from timestamps you can re-verify in thirty seconds, and worth actually checking rather than trusting on faith.
Tips
- Ask for a "source" column on the timeline unconditionally — an incident timeline without traceable sources is not something you should put your name on in a shared postmortem.
- Push the agent past generic action items ("add more monitoring") toward specific, ownable ones (which alarm, on which metric, owned by which team) — the first draft is often too vague to be actionable as written.
- Re-verify any specific quantitative claim in the draft (like a "would have caught it N minutes earlier" estimate) against the raw data before publishing — it's a checkable number, so check it.
Tips
Tips
- Run this four-phase shape (context → correlate → confirm/mitigate → document) as your standing incident runbook regardless of which agent client you're using — the tool calls and prompts translate directly across Claude Code, OpenCode, Gemini CLI, and Cursor.
- Insist on evidence (query strings, trace IDs, specific timestamps) behind every claim the agent makes, especially root-cause conclusions — speed of retrieval is the win here, not a shortcut around verification.
- Keep the whole incident in one continuous session so the final documentation step can cite real tool outputs rather than reconstructing the story from memory.
- Turn every postmortem action item into something specific and ownable before it leaves the draft stage — vague AI-generated suggestions are a common failure mode if you don't push back on the first pass.