This topic ties the module together with a single, complete incident from alert to postmortem, using Kubernetes MCP throughout. The scenario: an on-call alert fires for elevated error rates on checkout-service in the checkout-prod namespace, ten minutes after a routine deploy went out. Every command and tool call below reflects what actually happens in this kind of incident, not a simplified teaching example — including the parts where the first hypothesis is wrong.
Workflow Overview: From Alert to Stabilized Cluster
The shape of this workflow is deliberate and applies beyond this one scenario:
- Gather — pod status, events, and logs across the affected namespace and any adjacent ones sharing infrastructure (shared database, shared cache).
- Correlate — cross-reference the failure pattern against what changed recently: deploys, config, resource limits, dependency health.
- Act — execute the smallest safe fix that stabilizes the system (often a rollback, sometimes a scale-up, occasionally a config patch), with human approval on the actual mutating call.
- Verify and document — confirm the fix worked with fresh cluster state, then write the postmortem while the tool-call history is still fresh, using it as your timeline source.
AI agents accelerate steps 1 and 2 dramatically — that's where the tedious cross-referencing lives. Step 3 should always retain a human-approval gate on production. Step 4 is where having a clean, tool-call-driven audit trail pays for itself: your postmortem timeline can be built directly from the sequence of MCP calls and their timestamps rather than reconstructed from memory after the fact.
Tips
- Resist the urge to jump straight to step 3 because the first piece of evidence looks conclusive — the example in this topic shows why gathering broadly first matters.
- Treat the tool-call sequence itself as incident documentation — save the transcript, don't just remember the conclusion.
- Keep the human-approval gate on step 3 non-negotiable in production, regardless of how confident the agent's diagnosis sounds.
Step 1: Gathering Pod Status, Events, and Logs Across Namespaces
The alert only names a service and an elevated error rate — not a cause. Start broad within the affected namespace, then check the namespaces it depends on.
checkout-service in checkout-prod is showing elevated 5xx errors since
roughly 10 minutes ago. List all pods in checkout-prod that aren't
Running/Ready, and separately list pods in the shared postgres-prod
and redis-prod namespaces that back it.
- checkout-api-7d8f9c6b45-x2vqp: Ready=false, restarts=3, reason=CrashLoopBackOff
- checkout-api-7d8f9c6b45-k9m2l: Ready=true, restarts=0
- checkout-api-7d8f9c6b45-p4n7q: Ready=true, restarts=0
- checkout-worker-5c7d8f9a1-z3x8v: Ready=true, restarts=0
- all pods Running/Ready, 0 restarts
One of four checkout-api replicas is crash-looping; the dependency namespaces are clean. That already rules out a database or cache outage as the root cause and narrows this to something specific to the new pod or the recent deploy.
Get previous logs and recent events for checkout-api-7d8f9c6b45-x2vqp.
events:
- reason: BackOff
message: "Back-off restarting failed container checkout-api in pod checkout-api-7d8f9c6b45-x2vqp"
- reason: Unhealthy
message: "Liveness probe failed: Get \"http://10.4.2.18:8080/healthz\": dial tcp 10.4.2.18:8080: connect: connection refused"
panic: failed to initialize payment gateway client: invalid API key format
/app/internal/payments/gateway.go:58
That panic is specific and immediately actionable — it's not a generic crash, it's an explicit failure reading a credential. Before jumping to "the API key is wrong," check whether this is new (post-deploy) or pre-existing by comparing the crash-looping pod's image/start time against the healthy ones.
Compare the image tag and start time of the crashing pod against the
three healthy checkout-api pods.
- checkout-api-7d8f9c6b45-x2vqp: image=checkout-api:sha-9f3a21, startTime=10 min ago
- checkout-api-7d8f9c6b45-k9m2l: image=checkout-api:sha-9f3a21, startTime=10 min ago
- checkout-api-7d8f9c6b45-p4n7q: image=checkout-api:sha-9f3a21, startTime=10 min ago
Wait — all four pods are on the same image and same start time, meaning this isn't a one-bad-replica situation; it's the entire fleet running the new deploy, and only one has crashed so far. That's a materially different, more urgent situation than the earlier "one pod is unhealthy" read suggested: the other three are one restart away from hitting the same panic if the underlying cause is deterministic rather than a transient blip. This is exactly the kind of correction that broad gathering catches before you commit to a narrow, wrong action.
Tips
- Always check whether an unhealthy pod is an outlier or represents the whole fleet's current state — same image and start time across "healthy" pods doesn't mean they're safe, only that they haven't hit the failure yet.
- A specific panic message (naming the exact failure, like a credential format) is worth trusting over vaguer signals like a generic liveness probe failure — chase the panic first.
- Rule out shared dependencies (database, cache) early and cheaply — it's a fast check that meaningfully narrows the investigation.
Step 2: AI Root Cause Correlation Between Deploy, Config, and Resource Limits
With "new deploy, credential-related panic, whole fleet at risk" established, correlate against what actually changed.
Check the checkout-api Deployment's recent rollout history and its
current env/secret references for anything payment-gateway related.
kubectl rollout history deployment/checkout-api -n checkout-prod
REVISION CHANGE-CAUSE
41 deploy: bump image to sha-8a1b90
42 deploy: bump image to sha-9f3a21 (current)
env:
- name: PAYMENT_GATEWAY_API_KEY
valueFrom:
secretKeyRef:
name: payment-gateway-creds
key: api-key
The env var reference itself didn't change between revisions — same Secret name, same key. So the credential reference is fine; either the Secret's actual value changed independently, or the new code (sha-9f3a21) changed how it parses that value.
Check git log for the checkout-api repo between sha-8a1b90 and sha-9f3a21
for any changes to payment gateway client initialization.
git log --oneline sha-8a1b90..sha-9f3a21 -- internal/payments/
9f3a21 fix(payments): enforce strict API key prefix validation
There it is: revision 42 introduced stricter validation on the API key format, and the value currently stored in the payment-gateway-creds Secret doesn't satisfy the new check — likely a legitimate-but-differently-formatted key that the old, looser validation accepted. This is a code-and-config interaction, not a pure code bug and not a pure config bug: the new validation is arguably correct, but it shipped without confirming the production secret conformed to it.
Tips
- When an env/secret reference is unchanged across revisions but a credential-related error appears post-deploy, check the new code for validation or parsing changes before assuming the secret value itself needs rotating.
-git log <old-sha>..<new-sha> -- <path>scoped to the suspect subsystem is far faster than reading a full diff — ask the agent to scope it this way rather than reviewing the entire changeset.
- A "correct" code change (stricter validation) can still be the incident's proximate cause if it shipped without verifying production data/config compatibility — root cause and "whose fault" are different questions; focus on the former during triage.
Step 3: Executing a Safe Rollback and Writing the Postmortem
With three of four replicas still healthy but running code that will fail on their next restart, and the actual fix (correct the secret's key format, or relax/patch the validation) requiring a decision beyond immediate cluster action, the right call is an immediate rollback to stop the bleeding, followed by a proper fix through the normal deploy pipeline.
Propose rolling back checkout-api to revision 41. Show me the rollback
command — don't run it yet.
kubectl rollout undo deployment/checkout-api -n checkout-prod --to-revision=41
Approve explicitly, then have the agent execute and immediately verify:
Run that rollback, then confirm all checkout-api pods are Running/Ready
on the old image and restart counts have stopped climbing.
- checkout-api-8b3c7d2e91-a1f9k: image=sha-8a1b90, Ready=true, restarts=0
- checkout-api-8b3c7d2e91-b2g0l: image=sha-8a1b90, Ready=true, restarts=0
- checkout-api-8b3c7d2e91-c3h1m: image=sha-8a1b90, Ready=true, restarts=0
- checkout-api-8b3c7d2e91-d4i2n: image=sha-8a1b90, Ready=true, restarts=0
Four healthy replicas, old image, zero restarts. Confirm the error-rate signal externally too — the cluster looking healthy and the actual user-facing error rate recovering are two different checks, and only your monitoring/APM tool can confirm the second one. Don't close the incident on cluster state alone.
Check events across checkout-prod one more time for anything new in
the last 5 minutes, to make sure the rollback itself didn't trigger
a secondary issue.
Once confirmed stable, the postmortem timeline writes itself almost directly from the tool-call sequence:
Alert fired: 14:22 — checkout-service 5xx rate above threshold
14:24 — identified 1/4 checkout-api pods CrashLoopBackOff
14:26 — confirmed all 4 pods on same new image (sha-9f3a21), fleet-wide risk
14:29 — root cause: sha-9f3a21 introduced strict API key format validation;
production payment-gateway-creds secret value doesn't conform
14:33 — rolled back to revision 41 (sha-8a1b90)
14:35 — confirmed 4/4 pods healthy, error rate recovering per APM
14:40 — incident closed; follow-up ticket filed to fix secret format
and re-validate before re-deploying sha-9f3a21
The follow-up matters as much as the rollback — a rollback resolves the symptom, not the underlying mismatch between the new validation and the actual secret value. File that as a tracked action item, not a mental note, and make sure whoever re-attempts the deploy checks the secret's format before doing so.
Tips
- Always verify a rollback's success against both cluster state (pods healthy) and the actual external signal that triggered the alert (error rate, APM) — they can diverge, especially if the incident had a secondary contributing cause.
- Build the postmortem timeline directly from the tool-call sequence and its timestamps rather than reconstructing it from memory afterward — it's more accurate and takes less time.
- A rollback is a symptom fix, not a resolution — always file a concrete, owned follow-up for the actual root cause (in this case, the secret/validation mismatch) before considering the incident fully closed.
Tips
Tips
- Follow the gather-correlate-act-verify sequence deliberately, especially the "gather broadly before narrowing" step — this incident's initial read (one bad pod) would have led to the wrong fix if the agent hadn't checked whether the whole fleet shared the same risk.
- Keep the mutating action (rollback, scale, patch) behind explicit human approval even when the AI's root-cause analysis is thorough and convincing — approval is cheap, an unreviewed production mutation is not.
- Treat the AI-assisted tool-call transcript as your postmortem's primary source material — it's more precise than a reconstructed narrative and captures exact timestamps and evidence you'd otherwise have to dig back through logs to recover.