·

Real World Workflow Serverless Incident Investigation And Cost Optimization

Walk through a real production workflow that uses AWS MCP so your AI agent can manage cloud resources and investigate infrastructure end to end.

This topic walks through one complete, realistic incident end to end — a CloudWatch alarm firing on a production Lambda, through root cause analysis, a code fix, and a cost/reliability summary — using AWS MCP tools across the workflow. The scenario: checkout-order-processor, a Lambda behind an SQS queue, starts alarming on ApproximateAgeOfOldestMessage at 02:14 UTC. This is a composite of patterns from real incidents, not a single verbatim case, but every command and query below is exactly what you'd run against a real account.

Workflow Overview: From CloudWatch Alarm to Fix and Cost Report

The shape of this investigation is deliberate and worth internalizing as a repeatable pattern, not just for this incident:

  1. Pull alarm context — what fired, when, and what the alarm's own metric history shows.
  2. Correlate logs and traces — what was the function actually doing when the queue started backing up.
  3. Root cause across code, config, and IAM — is this a code bug, a config regression, or a permissions change.
  4. Apply the fix and verify — deploy, confirm the alarm clears, don't just assume it worked.
  5. Produce a cost and reliability summary — quantify the blast radius (SQS DLQ costs, reprocessing time, any duplicate-processing risk) for the postmortem.

Each step uses a different combination of AWS MCP tools, and the discipline of moving through them in order — rather than jumping straight to "just fix it" — is what makes the resulting postmortem defensible. An AI-assisted investigation that skips straight to a fix without documenting the alarm context and root cause evidence is exactly the kind of automation that makes a postmortem review harder, not easier.

Session goal: investigate the ApproximateAgeOfOldestMessage alarm on
checkout-order-processor's SQS queue, find root cause, propose a fix,
and prepare a cost/reliability summary for the incident channel.

Tips
- Treat this five-step shape as a template, not a one-off — the same sequence (alarm context → logs/traces → root cause → fix and verify → cost/reliability summary) applies to almost any serverless incident, not just SQS backlogs.
- Start the session by stating the full goal, including the postmortem deliverable, up front — it keeps the agent from stopping at "here's a fix" without producing the evidence trail you'll need afterward.

Step 1: Pulling Alarm Context, Logs, and Traces via MCP

Start with the alarm itself, not the logs — you need to know exactly what triggered and over what window before you go looking for a cause.

aws cloudwatch describe-alarms \
  --alarm-names "checkout-queue-age-high" \
  --profile dev-readonly

aws cloudwatch get-metric-data \
  --metric-data-queries '[{
    "Id": "queueAge",
    "MetricStat": {
      "Metric": {
        "Namespace": "AWS/SQS",
        "MetricName": "ApproximateAgeOfOldestMessage",
        "Dimensions": [{"Name": "QueueName", "Value": "checkout-orders-queue"}]
      },
      "Period": 60,
      "Stat": "Maximum"
    }
  }]' \
  --start-time 2026-08-21T01:30:00Z \
  --end-time 2026-08-21T03:00:00Z \
  --profile dev-readonly

With the exact alarm window established (01:58–02:31 UTC, message age climbing from 12s to 1,840s), pull the Lambda's own metrics for the same window to see whether it's an invocation failure, a throughput problem, or a downstream slowdown:

aws cloudwatch get-metric-data \
  --metric-data-queries '[
    {"Id":"invocations","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Invocations","Dimensions":[{"Name":"FunctionName","Value":"checkout-order-processor"}]},"Period":60,"Stat":"Sum"}},
    {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"checkout-order-processor"}]},"Period":60,"Stat":"Sum"}},
    {"Id":"duration","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Duration","Dimensions":[{"Name":"FunctionName","Value":"checkout-order-processor"}]},"Period":60,"Stat":"Average"}}
  ]' \
  --start-time 2026-08-21T01:30:00Z \
  --end-time 2026-08-21T03:00:00Z \
  --profile dev-readonly

Average duration climbing from 400ms to 28,000ms across the window, with invocation count dropping — a strong signal of a downstream dependency slowing down and Lambda concurrency/timeout limits choking throughput, not a code exception. Confirm with Logs Insights:

fields @timestamp, @message, @duration
| filter @duration > 20000
| sort @timestamp asc
| limit 30

And pull X-Ray traces for the slow invocations to see exactly which downstream call is the bottleneck, if X-Ray tracing is enabled on the function (it should be, for anything with an SLA):

aws xray get-trace-summaries \
  --start-time 2026-08-21T01:58:00Z \
  --end-time 2026-08-21T02:31:00Z \
  --filter-expression 'service("checkout-order-processor") { responsetime > 20 }' \
  --profile dev-readonly

aws xray batch-get-traces \
  --trace-ids 1-64e1a2b3-abcdef1234567890abcdef12 \
  --profile dev-readonly

The trace segment breakdown shows 27 of the 28 seconds spent in a single downstream call — the inventory reservation service. That's the concrete lead: this isn't a bug in checkout-order-processor itself, it's a slow dependency it doesn't handle gracefully.

Tips
- Always pull alarm metric history before logs — it establishes the precise time window and often the failure category (errors vs. latency vs. throttling) before you've read a single log line, which narrows every subsequent query.
- Enable X-Ray active tracing on any Lambda with a customer-facing SLA before you need it — reconstructing a downstream latency breakdown from logs alone, without trace segments, is possible but far slower and less precise.

Step 2: AI Root Cause Analysis Across Code, Config, and IAM

With the trace evidence pointing at the inventory reservation call, check the actual code path and its configured timeout — the classic serverless failure mode is a downstream call with no client-side timeout, so a slow dependency doesn't just slow you down, it holds the Lambda's own execution slot until the function's overall timeout kills it.

def reserve_inventory(order_id: str, items: list[dict]) -> dict:
    response = requests.post(
        f"{INVENTORY_SERVICE_URL}/reserve",
        json={"order_id": order_id, "items": items},
    )
    response.raise_for_status()
    return response.json()

No timeout= parameter on requests.post. Combined with the Lambda's own configured timeout:

aws lambda get-function-configuration \
  --function-name checkout-order-processor \
  --query "[Timeout,MemorySize,ReservedConcurrentExecutions]" \
  --profile dev-readonly

[30, 512, 20]

A 30-second Lambda timeout with an unbounded downstream call means every slow inventory-service request occupies a concurrency slot for up to 30 seconds. With ReservedConcurrentExecutions capped at 20, once ~20 requests are simultaneously stuck waiting on the slow dependency, no further messages get processed at all — which is exactly the queue-backup pattern the alarm caught. This is the root cause: not a code exception, a missing timeout combined with a concurrency ceiling that turns one slow dependency into a full processing stall.

Rule out an IAM or permissions change as a contributing factor before finalizing this, since a permissions change can sometimes cause a different service to start rejecting requests slowly rather than immediately (a common pattern with services that retry internally before returning a 403):

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=checkout-order-processor \
  --start-time 2026-08-20T00:00:00Z \
  --end-time 2026-08-21T02:00:00Z \
  --profile dev-readonly

No relevant IAM or configuration changes in the prior 24 hours — this rules out "someone changed a permission and inventory-service started slow-failing as a result" and confirms the root cause is purely the missing timeout plus concurrency ceiling.

Tips
- Always check ReservedConcurrentExecutions alongside function timeout when diagnosing a queue-backup incident — the interaction between the two, not either alone, is usually the actual mechanism turning a slow dependency into a full stall.
- Run a CloudTrail lookup for the affected resource across the prior 24–48 hours as a rule-out step, even when the evidence already points elsewhere — it's cheap insurance against attributing an incident to the wrong cause in the postmortem.

Step 3: Applying the Fix and Producing a Cost and Reliability Summary

The fix has two parts: bound the downstream call with an explicit timeout and retry policy, and raise (or at minimum, alert on) the concurrency ceiling so a legitimately slow-but-recovering dependency doesn't fully stall the queue.

def reserve_inventory(order_id: str, items: list[dict]) -> dict:
    response = requests.post(
        f"{INVENTORY_SERVICE_URL}/reserve",
        json={"order_id": order_id, "items": items},
        timeout=8,
    )
    response.raise_for_status()
    return response.json()

resource "aws_lambda_function" "checkout_order_processor" {
  # ...
  reserved_concurrent_executions = 40  # raised from 20, INC-2044
  timeout                        = 15  # lowered from 30 — fail fast, let SQS retry
}

Lowering the Lambda timeout alongside raising concurrency is deliberate: with the downstream call now bounded at 8 seconds, a 15-second function timeout gives enough margin for retries within the handler while failing fast enough that a genuinely stuck dependency releases the concurrency slot quickly, rather than holding it for the full 30 seconds as before.

terraform plan -target=aws_lambda_function.checkout_order_processor
terraform apply -target=aws_lambda_function.checkout_order_processor

Verify the alarm clears and doesn't just look temporarily better — watch the actual queue age metric recover, not just the absence of new alarm notifications:

aws cloudwatch get-metric-data \
  --metric-data-queries '[{
    "Id": "queueAge",
    "MetricStat": {
      "Metric": {
        "Namespace": "AWS/SQS",
        "MetricName": "ApproximateAgeOfOldestMessage",
        "Dimensions": [{"Name": "QueueName", "Value": "checkout-orders-queue"}]
      },
      "Period": 60,
      "Stat": "Maximum"
    }
  }]' \
  --start-time 2026-08-21T02:45:00Z \
  --end-time 2026-08-21T03:15:00Z \
  --profile dev-readonly

Finally, quantify the blast radius for the postmortem — messages that landed in the DLQ, any duplicate-processing risk from retried messages, and the marginal cost of the incident itself:

aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/checkout-orders-dlq \
  --attribute-names ApproximateNumberOfMessages \
  --profile dev-readonly

aws ce get-cost-and-usage \
  --time-period Start=2026-08-21,End=2026-08-22 \
  --granularity DAILY \
  --metrics "UnblendedCost" \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["AWS Lambda"]}}' \
  --profile dev-readonly

The incident summary in this case: 340 messages queued past SLA, 12 landed in the DLQ requiring manual reprocessing (checked for idempotency before replay — reserve_inventory is idempotent on order_id, confirmed by reading the handler, so replay was safe), 33 minutes of degraded checkout latency, and negligible direct AWS cost impact (Lambda's per-invocation cost is trivial; the real cost was customer-facing latency, not cloud spend). That distinction — negligible infra cost, real reliability cost — is exactly the kind of finding that a pure cost-dashboard view would miss, and exactly why this workflow pulls CloudWatch, X-Ray, IAM/CloudTrail, and Cost Explorer together rather than treating them as separate silos.

Tips
- Verify a fix by watching the actual metric recover over a real time window, not by the absence of new alarm notifications — an alarm can go quiet because the condition improved or because notifications are suppressed; check the underlying data.
- Confirm idempotency before replaying any DLQ messages — "safe to reprocess" is a claim you should verify by reading the actual handler logic (or an explicit idempotency key check), not assume because the messages look intact.

Tips

Tips
- Run every serverless incident through the same five-step shape — alarm context, logs/traces, root cause across code/config/IAM, fix and verify, cost/reliability summary — it produces a postmortem-ready evidence trail as a side effect of the investigation itself, rather than as separate work afterward.
- Check the interaction between Lambda timeout and reserved concurrency specifically whenever a queue-backed function alarms on age or backlog — it's a recurring root cause pattern, not a one-off, and MCP-driven inspection makes checking it in one session cheap enough to do by default.
- Distinguish infra cost impact from reliability impact explicitly in your summary — pulling Cost Explorer data as part of every incident report, even when the dollar amount is trivial, builds the habit of quantifying blast radius in whichever dimension actually matters for that incident.