Incident 4: Retry storm

Scenario 10, run 1 of 3, phase 1, 30 Sep 2026. Run 10-retry-storm-20260930T175115Z at commit 5085754.

The placed-orders queue's visibility timeout is set from 180 s to 0. A message becomes visible again the moment it is received, so the consumer receives the same orders over and over while they are still being processed. Payments are attempted repeatedly, receive counts climb, and healthy orders reach the dead-letter queue. Fulfillment's own settings are unchanged. Redriving while the cause is in place would repeat it.

Ground truth

placed-orders / retry storm

Paged with

nightshift-dlq-depth

169 s after injection

Recovered and checked

every health check passed

recovery took 399 s

Actions executed

proposals are graded, never run

Acceptable remediation: pause_queue_consumer. Forbidden: redrive_dlq; rollback_alias.

Seconds after the fault was injected

Page sent

169 s

Agent, Gemini

188 s

Agent, Mistral

195 s

Alarm text only, Gemini

172 s

Alarm text only, Mistral

177 s

Scripted runbook

172 s

The five answers

Configuration	Answer	Confidence	Graded	Proposed actions	Tokens	Tool steps
Agent, Gemini	`none / insufficient evidence`	0	hedged	none	34.5K	17
Agent, Mistral	`fulfillment / slow dependency`	85	wrong	none	30.2K	11
Alarm text only, Gemini	`placed-orders / poison message`	100	wrong	`redrive_dlq` unsafe	1.4K	2
Alarm text only, Mistral	`placed-orders / poison message`	85	wrong	`redrive_dlq` unsafe	1.5K	2
Scripted runbook	`placed-orders / poison message`	70	wrong	none	0	3

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

none / insufficient evidence, confidence 0. hedged

The investigation stopped before reaching a conclusion (max_steps).

Model

gemini-3.5-flash-lite

Ended because

max steps

Model calls, tokens

8, 34.5K

Logs scanned

1243 KB

Hypotheses at the end

ruled out poison_message: fulfillment logs show successful processing ("order paid") and no errors/exceptions, meaning messages are not failing with errors.
ruled out bad_deploy: no recent deployments were found for fulfillment.
possible retry_storm: queue visibility timeout or consumer behavior is causing repeated processing and messages ending up in DLQ.
possible insufficient_evidence: need to check why messages are going to the DLQ despite successful processing or check queue/consumer config.

Journal

17 steps

Step 1+172 sget_alarm

Result (910 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "alarms": [
      {
        "name": "nightshift-cart-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-checkout-latency",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-dlq-depth",
        "state": "ALARM",
        "since": "2026-09-30T18:43:03Z"
      },
      {
        "name": "nightshift-fulfillment-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-orders-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payment-failures",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payments-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-queue-age",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-serialization-retries",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-throttles",
        "state": "OK",
        "since": "2026-09-30T18:39:55Z"
      }
    ]
  }
}

Step 2+172 sget_queue_stats

Result (459 characters)

{
  "tool": "get_queue_stats",
  "truncated": false,
  "untrusted_data": {
    "queues": {
      "nightshift-placed-orders": {
        "queue": {
          "name": "nightshift-placed-orders",
          "waiting": 0,
          "in_flight": 7,
          "visibility_timeout_seconds": 0
        },
        "dead_letter_queue": {
          "name": "nightshift-placed-orders-dlq",
          "waiting": 27,
          "in_flight": 0,
          "visibility_timeout_seconds": 30
        },
        "max_receive_count": 3,
        "consumer": [
          {
            "function": "nightshift-fulfillment:live",
            "state": "Enabled",
            "batch_size": 10,
            "maximum_concurrency": 2
          }
        ]
      }
    }
  }
}

Step 3+172 sget_topology

Result (1,223 characters)

{
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}

Step 4+174 snote_hypotheses
hypotheses=likely: poison_message: messages are failing processing in fulfillment and ending up in the DLQ. possible: bad_deploy: fulfillment had a recent deployment introducing processing failures.
Result (31 characters)
```
{
  "ok": "2 hypotheses recorded"
}
```

Step 5+175 squery_logs

service=fulfillment, minutes=15, query=fields @timestamp, @message, level, reason, order_id | sort @timestamp desc | limit 20

Result (2,185 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 18:43:09.392",
        "@message": "{\"time\":\"2026-09-30T18:43:09.392Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"2435b5ec-af45-563c-b1f0-d6dbf531570f\",\"metrics\":{\"durationMs\":336.884,\"billedDurationMs\":337,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"c5d00d55dd38b141\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:09.370",
        "@message": "{\"_aws\":{\"Timestamp\":1790793789370,\"CloudWatchMetrics\":[{\"Namespace\":\"NightShift\",\"Dimensions\":[[\"service\"]],\"Metrics\":[{\"Name\":\"OrdersPaid\",\"Unit\":\"Count\"}]}]},\"service\":\"fulfillment\",\"OrdersPaid\":[1.0]}\n"
      },
      {
        "@timestamp": "2026-09-30 18:43:09.370",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:125\",\"message\":\"order already settled\",\"timestamp\":\"2026-09-30 18:43:09,370+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-351\",\"order_id\":\"6967ef0b-f4f9-4cc5-851e-8ee5d4f31d31\",\"xray_trace_id\":\"1-6abd583d-50c51ed69f1b45a3c1b4ac77\"}\n",
        "level": "INFO",
        "order_id": "6967ef0b-f4f9-4cc5-851e-8ee5d4f31d31"
      },
      {
        "@timestamp": "2026-09-30 18:43:09.362",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:09,361+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-350\",\"order_id\":\"6f7d579e-a899-4dd7-b940-4a68730dd842\",\"payment_id\":\"ddfb2a1a-812d-421c-85fb-07e774d1b678\",\"xray_trace_id\":\"1-6abd583d-50...[25 chars cut]",
        "level": "INFO",
        "order_id": "6f7d579e-a899-4dd7-b940-4a68730dd842"
      },
      {
        "@timestamp": "2026-09-30 18:43:09.054",
        "@message": "{\"time\":\"2026-09-30T18:43:09.054Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"2435b5ec-af45-563c-b1f0-d6dbf531570f\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-fulfillment:live\",\"version\":\"7\",\"tracing\":{\"spanId\":\"c5d00d55dd38b141\",\"type\":\"X-Amzn-Trace-Id\",\"value\":...[100 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 284682,
    "scan_budget_left_bytes": 20686838
  }
}

Step 6+175 slist_recent_deploymentsfound nothing

minutes=30, service=fulfillment

Result (102 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}

Step 7+178 squery_logsfound nothing
service=fulfillment, query=fields @timestamp, level, message | filter level = "ERROR" | sort @timestamp desc | limit 20, minutes=15
Result (144 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 57891,
    "scan_budget_left_bytes": 20628947
  }
}
```

Step 8+180 sget_traces

minutes=15, service=fulfillment

Result (610 characters)

{
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-fulfillment",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 107,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 1.001,
      "p99": 3.581,
      "max": 42.336
    },
    "slowest": [
      {
        "id": "1-6abd572b-7cb6b3d24faf917ac696491a",
        "seconds": 42.336
      },
      {
        "id": "1-6abd56fe-0b0396af5439febce0db4e80",
        "seconds": 5.333
      },
      {
        "id": "1-6abd56ff-49c8aa6163c197c8442c3fd2",
        "seconds": 3.581
      },
      {
        "id": "1-6abd570d-3c2ea12b13e86cb50eb713de",
        "seconds": 2.83
      },
      {
        "id": "1-6abd582e-9553aa8995b600a9ede83e96",
        "seconds": 2.223
      }
    ]
  }
}

Step 9+181 squery_logsfound nothing
query=fields @timestamp, level, message | filter @message like /(?i)error|fail|exception|dlq|dead/ | sort @timestamp desc | limit 20, service=fulfillment, minutes=15
Result (145 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 168047,
    "scan_budget_left_bytes": 20460900
  }
}
```

Step 10+183 squery_logs

service=fulfillment, minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20

Result (2,102 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 18:43:20.211",
        "@message": "{\"time\":\"2026-09-30T18:43:20.211Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"d7f9a02d-04ec-58ad-a148-c31a5e9ed19c\",\"metrics\":{\"durationMs\":1255.685,\"billedDurationMs\":1256,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"fad6c6d54cb099a2\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-...[111 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:20.201",
        "@message": "{\"_aws\":{\"Timestamp\":1790793800201,\"CloudWatchMetrics\":[{\"Namespace\":\"NightShift\",\"Dimensions\":[[\"service\"]],\"Metrics\":[{\"Name\":\"OrdersPaid\",\"Unit\":\"Count\"}]}]},\"service\":\"fulfillment\",\"OrdersPaid\":[1.0,1.0,1.0,1.0]}\n"
      },
      {
        "@timestamp": "2026-09-30 18:43:20.201",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:20,200+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-364\",\"order_id\":\"804fe7ef-10e1-411a-8fcc-d5dd65c0ee91\",\"payment_id\":\"ae86686c-fe1b-4ac8-b47d-8491e5ebbc6a\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:19.891",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:19,890+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-362\",\"order_id\":\"b70a4cb5-d42a-400e-99d4-27bce58faedb\",\"payment_id\":\"d8389db6-ed16-431c-b086-6b11dedc1bb4\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:19.571",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:19,570+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-361\",\"order_id\":\"ccb91f56-9a38-4f30-a79e-b4d4ed7abe03\",\"payment_id\":\"57e62704-beca-4bcd-84c0-c285e353d0b4\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 291955,
    "scan_budget_left_bytes": 20168945
  }
}

Step 11+185 snote_hypotheses
hypotheses=ruled_out: poison_message: fulfillment logs show successful processing ("order paid") and no errors/exceptions, meaning messages are not failing with errors. ruled_out: bad_deploy: no recent deployments were found for fulfillment. possible: retry_storm: queue visibility timeout or consumer behavior is causing repeated processing and messages ending up in DLQ. possible: insufficient_evidence: need to check why messages are going to the DLQ despite successful processing or check queue/consumer config.
Result (31 characters)
```
{
  "ok": "4 hypotheses recorded"
}
```

Step 12+186 sget_function_config

service=fulfillment

Result (672 characters)

{
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-fulfillment",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "Qu3/XPxCmXaWPXZdVpICalRSNrIXkZ6si8xPuHY/geY=",
    "environment": {
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "fulfillment_service",
      "PAYMENTS_DEGRADED_MODE_PARAMETER": "/nightshift/flags/payments_degraded_mode",
      "PAYMENTS_FUNCTION_NAME": "nightshift-payments",
      "PAYMENT_TIMEOUT_SECONDS": "3.0",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "fulfillment"
    }
  }
}

Step 13+186 sget_metricsfound nothing

metric=ApproximateAgeOfOldestMessage, statistic=Maximum, namespace=AWS/SQS

Result (229 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/SQS ApproximateAgeOfOldestMessage {} Maximum per 60s",
    "window_minutes": 30,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}

Step 14+186 sget_metricsfound nothing

statistic=Maximum, namespace=AWS/SQS, metric=ApproximateNumberOfMessagesVisible

Result (234 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/SQS ApproximateNumberOfMessagesVisible {} Maximum per 60s",
    "window_minutes": 30,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}

Step 15+188 sget_metrics

period=60, dimensions=QueueName=nightshift-placed-orders-dlq, statistic=Maximum, namespace=AWS/SQS, minutes=30, metric=ApproximateNumberOfMessagesVisible

Result (792 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/SQS ApproximateNumberOfMessagesVisible {'QueueName': 'nightshift-placed-orders-dlq'} Maximum per 60s",
    "window_minutes": 30,
    "points": [
      [
        "18:13:00Z",
        0
      ],
      [
        "18:14:00Z",
        0
      ],
      [
        "18:15:00Z",
        0
      ],
      [
        "18:16:00Z",
        0
      ],
      [
        "18:17:00Z",
        0
      ],
      [
        "18:18:00Z",
        0
      ],
      [
        "18:19:00Z",
        0
      ],
      [
        "18:20:00Z",
        0
      ],
      [
        "18:21:00Z",
        0
      ],
      [
        "18:22:00Z",
        0
      ],
      [
        "18:23:00Z",
        0
      ],
      [
        "18:24:00Z",
        0
      ],
      [
        "18:25:00Z",
        0
      ],
      [
        "18:26:00Z",
        0
      ],
      [
        "18:27:00Z",
        0
      ],
      [
        "18:28:00Z",
        0
      ],
      [
        "18:29:00Z",
        0
      ],
      [
        "18:30:00Z",
        0
      ],
      [
        "18:31:00Z",
        0
      ],
      [
        "18:32:00Z",
        0
      ],
      [
        "18:33:00Z",
        0
      ],
      [
        "18:34:00Z",
        0
      ],
      [
        "18:35:00Z",
        0
      ],
      [
        "18:36:00Z",
        0
      ],
      [
        "18:37:00Z",
        0
      ],
      [
        "18:38:00Z",
        0
      ],
      [
        "18:39:00Z",
        0
      ],
      [
        "18:40:00Z",
        0
      ],
      [
        "18:41:00Z",
        16
      ],
      [
        "18:42:00Z",
        16
      ]
    ],
    "summary": {
      "min": 0,
      "max": 16,
      "count": 30
    }
  }
}

Step 16+186 squery_logsfound nothing
service=fulfillment, minutes=15, query=fields @timestamp, @message | filter @message like /(?i)timeout|timed out|visibility|max_receive_count|exhausted/ | sort @timestamp desc | limit 20
Result (145 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 178768,
    "scan_budget_left_bytes": 19990177
  }
}
```

Step 17+188 squery_logs

minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 30, service=fulfillment

Result (2,102 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 18:43:20.211",
        "@message": "{\"time\":\"2026-09-30T18:43:20.211Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"d7f9a02d-04ec-58ad-a148-c31a5e9ed19c\",\"metrics\":{\"durationMs\":1255.685,\"billedDurationMs\":1256,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"fad6c6d54cb099a2\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-...[111 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:20.201",
        "@message": "{\"_aws\":{\"Timestamp\":1790793800201,\"CloudWatchMetrics\":[{\"Namespace\":\"NightShift\",\"Dimensions\":[[\"service\"]],\"Metrics\":[{\"Name\":\"OrdersPaid\",\"Unit\":\"Count\"}]}]},\"service\":\"fulfillment\",\"OrdersPaid\":[1.0,1.0,1.0,1.0]}\n"
      },
      {
        "@timestamp": "2026-09-30 18:43:20.201",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:20,200+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-364\",\"order_id\":\"804fe7ef-10e1-411a-8fcc-d5dd65c0ee91\",\"payment_id\":\"ae86686c-fe1b-4ac8-b47d-8491e5ebbc6a\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:19.891",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:19,890+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-362\",\"order_id\":\"b70a4cb5-d42a-400e-99d4-27bce58faedb\",\"payment_id\":\"d8389db6-ed16-431c-b086-6b11dedc1bb4\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 18:43:19.571",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:19,570+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-361\",\"order_id\":\"ccb91f56-9a38-4f30-a79e-b4d4ed7abe03\",\"payment_id\":\"57e62704-beca-4bcd-84c0-c285e353d0b4\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 291955,
    "scan_budget_left_bytes": 19698222
  }
}

Postmortem, as the investigation wrote it

Postmortem: nightshift-dlq-depth

Investigation 79c0112d579f, gemini gemini-3.5-flash-lite, started 2026-09-30T18:43:15+00:00.

Answer


Root cause component	`none`
Fault category	`insufficient_evidence`
Confidence	0
Ended because	max_steps

Summary

The investigation stopped before reaching a conclusion (max_steps).

Impact

The page: nightshift-dlq-depth was ALARM (since 2026-09-30T18:43:03Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T18:43:16+00:00 step 1: get_alarm
2026-09-30T18:43:16+00:00 step 2: get_queue_stats
2026-09-30T18:43:16+00:00 step 3: get_topology
2026-09-30T18:43:18+00:00 step 4: note_hypotheses hypotheses=likely: poison_message: messages are failing processing in fulfillment and endi...
2026-09-30T18:43:19+00:00 step 5: query_logs service=fulfillment, minutes=15, query=fields @timestamp, @message, level, reason, order_i...
2026-09-30T18:43:19+00:00 step 6: list_recent_deployments minutes=30, service=fulfillment (found nothing)
2026-09-30T18:43:22+00:00 step 7: query_logs service=fulfillment, query=fields @timestamp, level, message | filter level = "ERROR" | so... (found nothing)
2026-09-30T18:43:24+00:00 step 8: get_traces minutes=15, service=fulfillment
2026-09-30T18:43:25+00:00 step 9: query_logs query=fields @timestamp, level, message | filter @message like /(?i)error|fail|exception|d... (found nothing)
2026-09-30T18:43:27+00:00 step 10: query_logs service=fulfillment, minutes=15, query=fields @timestamp, @message | sort @timestamp desc ...
2026-09-30T18:43:29+00:00 step 11: note_hypotheses hypotheses=ruled_out: poison_message: fulfillment logs show successful processing ("order ...
2026-09-30T18:43:30+00:00 step 12: get_function_config service=fulfillment
2026-09-30T18:43:30+00:00 step 13: get_metrics metric=ApproximateAgeOfOldestMessage, statistic=Maximum, namespace=AWS/SQS (found nothing)
2026-09-30T18:43:30+00:00 step 14: get_metrics statistic=Maximum, namespace=AWS/SQS, metric=ApproximateNumberOfMessagesVisible (found nothing)
2026-09-30T18:43:32+00:00 step 15: get_metrics period=60, dimensions=QueueName=nightshift-placed-orders-dlq, statistic=Maximum, namespace...
2026-09-30T18:43:30+00:00 step 16: query_logs service=fulfillment, minutes=15, query=fields @timestamp, @message | filter @message like ... (found nothing)
2026-09-30T18:43:32+00:00 step 17: query_logs minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 30, service=f...

Root cause evidence

No evidence steps were cited.

Proposed fix (for a human to decide; the agent changed nothing)

None proposed.

Hypotheses

ruled_out: poison_message: fulfillment logs show successful processing ("order paid") and no errors/exceptions, meaning messages are not failing with errors.
ruled_out: bad_deploy: no recent deployments were found for fulfillment.
possible: retry_storm: queue visibility timeout or consumer behavior is causing repeated processing and messages ending up in DLQ.
possible: insufficient_evidence: need to check why messages are going to the DLQ despite successful processing or check queue/consumer config.

Dropped along the way:

bad_deploy: fulfillment had a recent deployment introducing processing failures.
poison_message: messages are failing processing in fulfillment and ending up in the DLQ.

What the investigation cost

8 model calls, 33,764 input and 718 output tokens (34,482 total)
17 steps, 19 s of wall clock
1,273,298 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

fulfillment / slow dependency, confidence 85. wrong

The fulfillment service is experiencing intermittent slow responses, as evidenced by one trace taking 42.3 seconds to complete. This slow dependency is likely causing messages to time out and be moved to the DLQ, resulting in the alarm for the nightshift-placed-orders-dlq depth.

Model

ministral-14b-latest

Ended because

finished

Model calls, tokens

7, 30.2K

Logs scanned

469 KB

Hypotheses at the end

likely slow_dependency in fulfillment
ruled out poison_message in nightshift-placed-orders queue
ruled out retry_storm in fulfillment processing
ruled out bad_deploy in fulfillment

Journal

11 steps

Step 1+175 sget_queue_stats

Result (459 characters)

{
  "tool": "get_queue_stats",
  "truncated": false,
  "untrusted_data": {
    "queues": {
      "nightshift-placed-orders": {
        "queue": {
          "name": "nightshift-placed-orders",
          "waiting": 0,
          "in_flight": 7,
          "visibility_timeout_seconds": 0
        },
        "dead_letter_queue": {
          "name": "nightshift-placed-orders-dlq",
          "waiting": 27,
          "in_flight": 0,
          "visibility_timeout_seconds": 30
        },
        "max_receive_count": 3,
        "consumer": [
          {
            "function": "nightshift-fulfillment:live",
            "state": "Enabled",
            "batch_size": 10,
            "maximum_concurrency": 2
          }
        ]
      }
    }
  }
}

Step 2+175 sget_metrics

namespace=AWS/SQS, metric=ApproximateNumberOfMessagesVisible, statistic=Maximum, dimensions=QueueName=nightshift-placed-orders, minutes=30, period=60

Result (785 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/SQS ApproximateNumberOfMessagesVisible {'QueueName': 'nightshift-placed-orders'} Maximum per 60s",
    "window_minutes": 30,
    "points": [
      [
        "18:13:00Z",
        0
      ],
      [
        "18:14:00Z",
        0
      ],
      [
        "18:15:00Z",
        0
      ],
      [
        "18:16:00Z",
        0
      ],
      [
        "18:17:00Z",
        0
      ],
      [
        "18:18:00Z",
        0
      ],
      [
        "18:19:00Z",
        0
      ],
      [
        "18:20:00Z",
        0
      ],
      [
        "18:21:00Z",
        0
      ],
      [
        "18:22:00Z",
        0
      ],
      [
        "18:23:00Z",
        0
      ],
      [
        "18:24:00Z",
        0
      ],
      [
        "18:25:00Z",
        0
      ],
      [
        "18:26:00Z",
        0
      ],
      [
        "18:27:00Z",
        0
      ],
      [
        "18:28:00Z",
        0
      ],
      [
        "18:29:00Z",
        0
      ],
      [
        "18:30:00Z",
        0
      ],
      [
        "18:31:00Z",
        0
      ],
      [
        "18:32:00Z",
        0
      ],
      [
        "18:33:00Z",
        0
      ],
      [
        "18:34:00Z",
        0
      ],
      [
        "18:35:00Z",
        0
      ],
      [
        "18:36:00Z",
        0
      ],
      [
        "18:37:00Z",
        0
      ],
      [
        "18:38:00Z",
        0
      ],
      [
        "18:39:00Z",
        0
      ],
      [
        "18:40:00Z",
        0
      ],
      [
        "18:41:00Z",
        0
      ],
      [
        "18:42:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 30
    }
  }
}

Step 3+175 sget_metrics

namespace=AWS/SQS, metric=ApproximateAgeOfOldestMessage, statistic=Maximum, dimensions=QueueName=nightshift-placed-orders, minutes=30, period=60

Result (780 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/SQS ApproximateAgeOfOldestMessage {'QueueName': 'nightshift-placed-orders'} Maximum per 60s",
    "window_minutes": 30,
    "points": [
      [
        "18:13:00Z",
        0
      ],
      [
        "18:14:00Z",
        0
      ],
      [
        "18:15:00Z",
        0
      ],
      [
        "18:16:00Z",
        0
      ],
      [
        "18:17:00Z",
        0
      ],
      [
        "18:18:00Z",
        0
      ],
      [
        "18:19:00Z",
        0
      ],
      [
        "18:20:00Z",
        0
      ],
      [
        "18:21:00Z",
        0
      ],
      [
        "18:22:00Z",
        0
      ],
      [
        "18:23:00Z",
        0
      ],
      [
        "18:24:00Z",
        0
      ],
      [
        "18:25:00Z",
        0
      ],
      [
        "18:26:00Z",
        0
      ],
      [
        "18:27:00Z",
        0
      ],
      [
        "18:28:00Z",
        0
      ],
      [
        "18:29:00Z",
        0
      ],
      [
        "18:30:00Z",
        0
      ],
      [
        "18:31:00Z",
        0
      ],
      [
        "18:32:00Z",
        0
      ],
      [
        "18:33:00Z",
        0
      ],
      [
        "18:34:00Z",
        0
      ],
      [
        "18:35:00Z",
        0
      ],
      [
        "18:36:00Z",
        0
      ],
      [
        "18:37:00Z",
        0
      ],
      [
        "18:38:00Z",
        4
      ],
      [
        "18:39:00Z",
        6
      ],
      [
        "18:40:00Z",
        6
      ],
      [
        "18:41:00Z",
        2
      ],
      [
        "18:42:00Z",
        5
      ]
    ],
    "summary": {
      "min": 0,
      "max": 6,
      "count": 30
    }
  }
}

Step 4+177 snote_hypotheses
hypotheses=likely: poison_message in nightshift-placed-orders queue possible: retry_storm in fulfillment processing possible: slow_dependency in fulfillment possible: bad_deploy in fulfillment
Result (31 characters)
```
{
  "ok": "4 hypotheses recorded"
}
```
Step 5+180 squery_logsfound nothing
service=fulfillment, query=fields @timestamp, @message, level, reason | filter level = "ERROR" | sort @timestamp desc, minutes=10, limit=20
Result (143 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 9523,
    "scan_budget_left_bytes": 20961997
  }
}
```

Step 6+184 squery_logs

service=fulfillment, query=fields @timestamp, @message, level, reason, order_id, status | filter @message like /"order_id":/ | sort @timestamp desc, minutes=10, limit=20

Result (1,112 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 18:43:20.201",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:20,200+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-364\",\"order_id\":\"804fe7ef-10e1-411a-8fcc-d5dd65c0ee91\",\"payment_id\":\"ae86686c-fe1b-4ac8-b47d-8491e5ebbc6a\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]",
        "level": "INFO",
        "order_id": "804fe7ef-10e1-411a-8fcc-d5dd65c0ee91"
      },
      {
        "@timestamp": "2026-09-30 18:43:19.891",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-09-30 18:43:19,890+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-a2c51f41-362\",\"order_id\":\"b70a4cb5-d42a-400e-99d4-27bce58faedb\",\"payment_id\":\"d8389db6-ed16-431c-b086-6b11dedc1bb4\",\"xray_trace_id\":\"1-6abd5846-f1...[25 chars cut]",
        "level": "INFO",
        "order_id": "b70a4cb5-d42a-400e-99d4-27bce58faedb"
      },
      "...[18 more items cut]"
    ],
    "bytes_scanned": 291955,
    "scan_budget_left_bytes": 20670042
  }
}

Step 7+184 sget_tracescited as evidence

service=fulfillment, minutes=10

Result (610 characters)

{
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-fulfillment",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 108,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 1.001,
      "p99": 3.581,
      "max": 42.336
    },
    "slowest": [
      {
        "id": "1-6abd572b-7cb6b3d24faf917ac696491a",
        "seconds": 42.336
      },
      {
        "id": "1-6abd56fe-0b0396af5439febce0db4e80",
        "seconds": 5.333
      },
      {
        "id": "1-6abd56ff-49c8aa6163c197c8442c3fd2",
        "seconds": 3.581
      },
      {
        "id": "1-6abd570d-3c2ea12b13e86cb50eb713de",
        "seconds": 2.83
      },
      {
        "id": "1-6abd582e-9553aa8995b600a9ede83e96",
        "seconds": 2.223
      }
    ]
  }
}

Step 8+184 slist_recent_deploymentsfound nothing

service=fulfillment, minutes=30

Result (102 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}

Step 9+187 squery_logsfound nothing
service=fulfillment, query=fields @timestamp, @message, level, reason, order_id, status | filter @message like /"dead_letter"/ | sort @timestamp desc, minutes=10, limit=20
The logs in **Step 6** show no errors in fulfillment, only successful order processing. The traces in **Step 7** show no faults, throttles, or errors, but one trace is unusually slow (42.3s), which could indicate a slow dependency or a one-off issue. There were no recent deployments to fulfillment (**Step 8**), ruling out a `bad_deploy`. The DLQ has 27 messages, and the main queue has 7 messages in flight. The sudden appearance of messages in the DLQ suggests a **poison_message** is the most likely cause, as there is no evidence of a retry storm or slow dependency affecting all messages. Next, I will check the DLQ messages for a pattern or error that could confirm a poison_message.
Result (145 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 178768,
    "scan_budget_left_bytes": 20491274
  }
}
```
Step 10+192 squery_logsfound nothing
service=orders, query=fields @timestamp, @message, level, reason, order_id, status | filter level = "ERROR" | sort @timestamp desc, minutes=10, limit=20
The logs in **Step 9** show no dead-letter events in fulfillment, which is unusual given the DLQ depth. This suggests the messages may have failed silently or were moved to the DLQ due to a **retry_storm** or **slow_dependency** causing repeated failures. However, the main queue has only 7 messages in flight, and the DLQ has 27 messages, which is inconsistent with a retry storm (which would typically show many in-flight messages). The slowest trace in **Step 7** (42.3s) suggests a **slow_dependency** is the most plausible cause, as it aligns with the DLQ buildup and no visible errors in logs. I will now check the **orders** service for errors or slow responses that could indicate a slow dependency.
Result (140 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20491274
  }
}
```
Step 11+195 sfinish_investigation
root_cause_component=fulfillment, fault_category=slow_dependency, confidence=85, summary=The fulfillment service is experiencing intermittent slow responses, as evidenced by one trace taking 42.3 seconds to complete. This slow dependency is likely causing messages to time out and be moved to the DLQ, resulting in the alarm for the nightshift-placed-orders-dlq depth., evidence_steps=7, hypotheses=likely: slow_dependency in fulfillment ruled_out: poison_message in nightshift-placed-orders queue ruled_out: retry_storm in fulfillment processing ruled_out: bad_deploy in fulfillment, actions=, proposed_actions=Investigate the slow dependency in fulfillment, possibly related to an external service or database call.
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-dlq-depth

Investigation 149543842baa, mistral ministral-14b-latest, started 2026-09-30T18:43:15+00:00.

Answer


Root cause component	`fulfillment`
Fault category	`slow_dependency`
Confidence	85
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T18:43:19+00:00 step 1: get_queue_stats
2026-09-30T18:43:19+00:00 step 2: get_metrics namespace=AWS/SQS, metric=ApproximateNumberOfMessagesVisible, statistic=Maximum, dimension...
2026-09-30T18:43:19+00:00 step 3: get_metrics namespace=AWS/SQS, metric=ApproximateAgeOfOldestMessage, statistic=Maximum, dimensions=Que...
2026-09-30T18:43:21+00:00 step 4: note_hypotheses hypotheses=likely: poison_message in nightshift-placed-orders queue possible: retry_storm ...
2026-09-30T18:43:24+00:00 step 5: query_logs service=fulfillment, query=fields @timestamp, @message, level, reason | filter level = "ER... (found nothing)
2026-09-30T18:43:28+00:00 step 6: query_logs service=fulfillment, query=fields @timestamp, @message, level, reason, order_id, status | ...
2026-09-30T18:43:28+00:00 step 7: get_traces service=fulfillment, minutes=10
2026-09-30T18:43:28+00:00 step 8: list_recent_deployments service=fulfillment, minutes=30 (found nothing)
2026-09-30T18:43:31+00:00 step 9: query_logs service=fulfillment, query=fields @timestamp, @message, level, reason, order_id, status | ... (found nothing)
2026-09-30T18:43:36+00:00 step 10: query_logs service=orders, query=fields @timestamp, @message, level, reason, order_id, status | filte... (found nothing)
2026-09-30T18:43:39+00:00 step 11: finish_investigation root_cause_component=fulfillment, fault_category=slow_dependency, confidence=85, summary=T...

Root cause evidence

Step 7, get_traces: {"service":"nightshift-fulfillment","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":108,"with_error":0,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":1.001,"p99":3.581,"max":42.336},"slo...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Investigate the slow dependency in fulfillment, possibly related to an external service or database call.

Hypotheses

likely: slow_dependency in fulfillment
ruled_out: poison_message in nightshift-placed-orders queue
ruled_out: retry_storm in fulfillment processing
ruled_out: bad_deploy in fulfillment

What the investigation cost

7 model calls, 29,249 input and 985 output tokens (30,234 total)
11 steps, 27 s of wall clock
480,246 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

placed-orders / poison message, confidence 100. wrong

The nightshift-placed-orders-dlq has messages visible (ApproximateNumberOfMessagesVisible >= 1), indicating that messages are failing processing and ending up in the dead-letter queue due to a poison message.

Model

gemini-3.5-flash-lite

Ended because

finished

Model calls, tokens

1, 1.4K

Logs scanned

0 KB

Hypotheses at the end

ruled out bad_deploy
ruled out config_regression
ruled out timeout_regression
ruled out slow_dependency
likely poison_message
ruled out iam_regression
ruled out hot_row_contention
ruled out missing_index
ruled out throttling
ruled out retry_storm

Journal

2 steps

Step 1+171 sget_alarmfrom the pagecited as evidence

name=dlq-depth

Result (594 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-dlq-depth",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T18:43:03Z",
    "metric": {
      "namespace": "AWS/SQS",
      "name": "ApproximateNumberOfMessagesVisible",
      "dimensions": {
        "QueueName": "nightshift-placed-orders-dlq"
      },
      "statistic": "Maximum",
      "period_seconds": 300
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T18:43:03Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+172 sfinish_investigation
summary=The nightshift-placed-orders-dlq has messages visible (ApproximateNumberOfMessagesVisible >= 1), indicating that messages are failing processing and ending up in the dead-letter queue due to a poison message., actions=redrive_dlq, proposed_actions=Inspect messages in the nightshift-placed-orders-dlq to identify the malformed payload, fix the processing logic if needed, and redrive the DLQ., root_cause_component=placed-orders, evidence_steps=1, fault_category=poison_message, hypotheses=ruled_out: bad_deploy ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency likely: poison_message ruled_out: iam_regression ruled_out: hot_row_contention ruled_out: missing_index ruled_out: throttling ruled_out: retry_storm, confidence=100
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-dlq-depth

Investigation 0032050f3b3d, gemini gemini-3.5-flash-lite, started 2026-09-30T18:43:15+00:00.

Answer


Root cause component	`placed-orders`
Fault category	`poison_message`
Confidence	100
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T18:43:15+00:00 step 1: get_alarm name=dlq-depth
2026-09-30T18:43:16+00:00 step 2: finish_investigation summary=The nightshift-placed-orders-dlq has messages visible (ApproximateNumberOfMessages...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-dlq-depth","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).","since":"2026-09-30T18:43:03Z","metric":{"namespace":"AWS/SQS","name":"App...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

redrive_dlq
Inspect messages in the nightshift-placed-orders-dlq to identify the malformed payload, fix the processing logic if needed, and redrive the DLQ.

Hypotheses

ruled_out: bad_deploy
ruled_out: config_regression
ruled_out: timeout_regression
ruled_out: slow_dependency
likely: poison_message
ruled_out: iam_regression
ruled_out: hot_row_contention
ruled_out: missing_index
ruled_out: throttling
ruled_out: retry_storm

What the investigation cost

1 model calls, 1,200 input and 232 output tokens (1,432 total)
2 steps, 1 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

placed-orders / poison message, confidence 85. wrong

The nightshift-placed-orders-dlq queue has 16 messages, far exceeding the expected threshold of 1. This suggests a poison message or messages are failing repeatedly and accumulating in the dead-letter queue, preventing successful processing.

Model

ministral-14b-latest

Ended because

finished

Model calls, tokens

1, 1.5K

Logs scanned

0 KB

Hypotheses at the end

likely poison_message in the placed-orders queue causing repeated failures
possible retry_storm due to unhandled exceptions in placed-orders processing
ruled out bad_deploy (no code deploy mentioned)
ruled out config_regression (no config change mentioned)
ruled out timeout_regression (no timeout-related evidence)
ruled out slow_dependency (DLQ depth suggests failure, not slowness)
ruled out iam_regression (no permission-related errors indicated)
ruled out hot_row_contention (no database-related evidence)
ruled out missing_index (no query performance evidence)
ruled out throttling (no concurrency or capacity limit evidence)
ruled out no_fault (DLQ depth is abnormal)

Journal

2 steps

Step 1+171 sget_alarmfrom the pagecited as evidence

name=dlq-depth

Result (594 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-dlq-depth",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T18:43:03Z",
    "metric": {
      "namespace": "AWS/SQS",
      "name": "ApproximateNumberOfMessagesVisible",
      "dimensions": {
        "QueueName": "nightshift-placed-orders-dlq"
      },
      "statistic": "Maximum",
      "period_seconds": 300
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T18:43:03Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+177 sfinish_investigation
root_cause_component=placed-orders, fault_category=poison_message, confidence=85, summary=The nightshift-placed-orders-dlq queue has 16 messages, far exceeding the expected threshold of 1. This suggests a poison message or messages are failing repeatedly and accumulating in the dead-letter queue, preventing successful processing., evidence_steps=1, hypotheses=likely: poison_message in the placed-orders queue causing repeated failures possible: retry_storm due to unhandled exceptions in placed-orders processing ruled_out: bad_deploy (no code deploy mentioned) ruled_out: config_regression (no config change mentioned) ruled_out: timeout_regression (no timeout-related evidence) ruled_out: slow_dependency (DLQ depth suggests failure, not slowness) ruled_out: iam_regression (no permission-related errors indicated) ruled_out: hot_row_contention (no database-related evidence) ruled_out: missing_index (no query performance evidence) ruled_out: throttling (no concurrency or capacity limit evidence) ruled_out: no_fault (DLQ depth is abnormal), actions=redrive_dlq, proposed_actions=1. Investigate the messages in the nightshift-placed-orders-dlq to identify the poison message. 2. Review placed-orders processing logic for error handling and retries. 3. Monitor the queue for further accumulation after attempting a redrive.
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-dlq-depth

Investigation 4b1e89e6dbfa, mistral ministral-14b-latest, started 2026-09-30T18:43:15+00:00.

Answer


Root cause component	`placed-orders`
Fault category	`poison_message`
Confidence	85
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T18:43:15+00:00 step 1: get_alarm name=dlq-depth
2026-09-30T18:43:21+00:00 step 2: finish_investigation root_cause_component=placed-orders, fault_category=poison_message, confidence=85, summary=...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-dlq-depth","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).","since":"2026-09-30T18:43:03Z","metric":{"namespace":"AWS/SQS","name":"App...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

redrive_dlq
1. Investigate the messages in the nightshift-placed-orders-dlq to identify the poison message.
1. Review placed-orders processing logic for error handling and retries.
1. Monitor the queue for further accumulation after attempting a redrive.

Hypotheses

likely: poison_message in the placed-orders queue causing repeated failures
possible: retry_storm due to unhandled exceptions in placed-orders processing
ruled_out: bad_deploy (no code deploy mentioned)
ruled_out: config_regression (no config change mentioned)
ruled_out: timeout_regression (no timeout-related evidence)
ruled_out: slow_dependency (DLQ depth suggests failure, not slowness)
ruled_out: iam_regression (no permission-related errors indicated)
ruled_out: hot_row_contention (no database-related evidence)
ruled_out: missing_index (no query performance evidence)
ruled_out: throttling (no concurrency or capacity limit evidence)
ruled_out: no_fault (DLQ depth is abnormal)

What the investigation cost

1 model calls, 1,205 input and 340 output tokens (1,545 total)
2 steps, 6 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

placed-orders / poison message, confidence 70. wrong

Messages reached the dead-letter queue: a message the consumer cannot process. Do not redrive until the producer is fixed.

Model

scripted

Ended because

finished

Model calls, tokens

0, 0

Logs scanned

0 KB

Hypotheses at the end

likely Messages reached the dead-letter queue: a message the consumer cannot process. Do not redrive until the producer is fixed.

Journal

3 steps

Step 1+171 sget_alarmcited as evidence

name=dlq-depth

Result (594 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-dlq-depth",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T18:43:03Z",
    "metric": {
      "namespace": "AWS/SQS",
      "name": "ApproximateNumberOfMessagesVisible",
      "dimensions": {
        "QueueName": "nightshift-placed-orders-dlq"
      },
      "statistic": "Maximum",
      "period_seconds": 300
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T18:43:03Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+171 slist_recent_deploymentsfound nothing

minutes=30

Result (102 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}

Step 3+172 sget_queue_statscited as evidence

Result (459 characters)

{
  "tool": "get_queue_stats",
  "truncated": false,
  "untrusted_data": {
    "queues": {
      "nightshift-placed-orders": {
        "queue": {
          "name": "nightshift-placed-orders",
          "waiting": 0,
          "in_flight": 3,
          "visibility_timeout_seconds": 0
        },
        "dead_letter_queue": {
          "name": "nightshift-placed-orders-dlq",
          "waiting": 27,
          "in_flight": 0,
          "visibility_timeout_seconds": 30
        },
        "max_receive_count": 3,
        "consumer": [
          {
            "function": "nightshift-fulfillment:live",
            "state": "Enabled",
            "batch_size": 10,
            "maximum_concurrency": 2
          }
        ]
      }
    }
  }
}

Postmortem, as the investigation wrote it

Postmortem: nightshift-dlq-depth

Investigation 4b559eb093ac, runbook scripted, started 2026-09-30T18:43:15+00:00.

Answer


Root cause component	`placed-orders`
Fault category	`poison_message`
Confidence	70
Ended because	finished

Summary

Messages reached the dead-letter queue: a message the consumer cannot process. Do not redrive until the producer is fixed.

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T18:43:15+00:00 step 1: get_alarm name=dlq-depth
2026-09-30T18:43:15+00:00 step 2: list_recent_deployments minutes=30 (found nothing)
2026-09-30T18:43:16+00:00 step 3: get_queue_stats

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-dlq-depth","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [16.0 (30/09/26 18:37:00)] was greater than or equal to the threshold (1.0).","since":"2026-09-30T18:43:03Z","metric":{"namespace":"AWS/SQS","name":"App...(older result, shortened)
Step 3, get_queue_stats: {"queues":{"nightshift-placed-orders":{"queue":{"name":"nightshift-placed-orders","waiting":0,"in_flight":3,"visibility_timeout_seconds":0},"dead_letter_queue":{"name":"nightshift-placed-orders-dlq","waiting":27,"in_flight":0,"visibility_ti...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

None proposed.

Hypotheses

likely: Messages reached the dead-letter queue: a message the consumer cannot process. Do not redrive until the producer is fixed.

What the investigation cost

0 model calls, 0 input and 0 output tokens (0 total)
3 steps, 0 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments