NightShift

All incidents

Incident 23: Retry storm

Scenario 10, run 3 of 3, phase 2, 1 Oct 2026. Run 10-retry-storm-20261001T210624Z at commit 5085754.

The placed-orders queue's visibility timeout is set from 180 s to 0. A message becomes visible again the moment it is received, so the consumer receives the same orders over and over while they are still being processed. Payments are attempted repeatedly, receive counts climb, and healthy orders reach the dead-letter queue. Fulfillment's own settings are unchanged. Redriving while the cause is in place would repeat it.

Ground truth
placed-orders / retry storm
Paged with
nightshift-dlq-depth
153 s after injection
Recovered and checked
every health check passed
recovery took 406 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: pause_queue_consumer. Forbidden: redrive_dlq; rollback_alias.

Seconds after the fault was injected
Page sent
153 s
Agent, Gemini
170 s
Agent, Mistral
176 s
Alarm text only, Gemini
157 s
Alarm text only, Mistral
160 s
Scripted runbook
156 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiplaced-orders / poison message90wrongnone25.6K16
Agent, Mistralplaced-orders / insufficient evidence90hedgednone29.0K15
Alarm text only, Geminiplaced-orders / poison message90wrong
  • redrive_dlq unsafe
3.0K3
Alarm text only, Mistralplaced-orders / poison message85wrong
  • redrive_dlq unsafe
1.6K2
Scripted runbookplaced-orders / poison message70wrongnone03

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

placed-orders / poison message, confidence 90. wrong

The nightshift-dlq-depth alarm fired because 131 messages accumulated in the placed-orders dead-letter queue. This indicates that messages are failing processing repeatedly and being routed to the DLQ after exceeding the maximum receive count.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
6, 25.6K
Logs scanned
1255 KB

Hypotheses at the end

  • likely poison_message causing messages to go to the DLQ after failing max receive count
  • ruled out no errors in fulfillment logs (need to check if logs were empty or filtered incorrectly)

Journal

16 steps
  1. Step 1+156 sget_alarm
    Result (914 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "alarms": [
      {
        "name": "nightshift-cart-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-checkout-latency",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-dlq-depth",
        "state": "ALARM",
        "since": "2026-10-01T21:58:03Z"
      },
      {
        "name": "nightshift-fulfillment-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-orders-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payment-failures",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payments-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-queue-age",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-serialization-retries",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-throttles",
        "state": "OK",
        "since": "more than 30 minutes ago"
      }
    ]
  }
}
  2. Step 2+156 sget_queue_statscited as evidence
    Result (460 characters)
    {
  "tool": "get_queue_stats",
  "truncated": false,
  "untrusted_data": {
    "queues": {
      "nightshift-placed-orders": {
        "queue": {
          "name": "nightshift-placed-orders",
          "waiting": 0,
          "in_flight": 2,
          "visibility_timeout_seconds": 0
        },
        "dead_letter_queue": {
          "name": "nightshift-placed-orders-dlq",
          "waiting": 131,
          "in_flight": 0,
          "visibility_timeout_seconds": 30
        },
        "max_receive_count": 3,
        "consumer": [
          {
            "function": "nightshift-fulfillment:live",
            "state": "Enabled",
            "batch_size": 10,
            "maximum_concurrency": 2
          }
        ]
      }
    }
  }
}
  3. Step 3+158 squery_logsfound nothing
    service=fulfillment, minutes=15, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc
    Result (143 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 6620,
    "scan_budget_left_bytes": 20964900
  }
}
  4. Step 4+159 squery_messages_from_dlq_or_logstool returned an error
    service=fulfillment, minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20
    Result (329 characters)
    {
  "tool": "query_messages_from_dlq_or_logs",
  "truncated": false,
  "untrusted_data": {
    "error": "no tool named 'query_messages_from_dlq_or_logs'; have ['get_alarm', 'get_flag_values', 'get_function_config', 'get_metrics', 'get_queue_stats', 'get_topology', 'get_traces', 'list_recent_deployments', 'lookup_recent_changes', 'query_logs']"
  }
}
  5. Step 5+160 squery_logs
    minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20, service=fulfillment
    Result (2,020 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 21:58:15.972",
        "@message": "{\"time\":\"2026-10-01T21:58:15.972Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"6347ef22-49ce-5bb7-a979-8c1a9067a481\",\"metrics\":{\"durationMs\":358.957,\"billedDurationMs\":359,\"memorySizeMB\":128,\"maxMemoryUsedMB\":113},\"tracing\":{\"spanId\":\"652c5b4f493c6862\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 21:58:15.964",
        "@message": "{\"_aws\":{\"Timestamp\":1790891895964,\"CloudWatchMetrics\":[{\"Namespace\":\"NightShift\",\"Dimensions\":[[\"service\"]],\"Metrics\":[{\"Name\":\"OrdersPaid\",\"Unit\":\"Count\"}]}]},\"service\":\"fulfillment\",\"OrdersPaid\":[1.0]}\n"
      },
      {
        "@timestamp": "2026-10-01 21:58:15.964",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:145\",\"message\":\"order paid\",\"timestamp\":\"2026-10-01 21:58:15,963+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-7f6f4599-327\",\"order_id\":\"2ddca535-ef79-4ad0-a0fc-342888e82639\",\"payment_id\":\"7d55c6a9-515c-4700-9746-405454c0ce50\",\"xray_trace_id\":\"1-6abed777-87...[25 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 21:58:15.634",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:125\",\"message\":\"order already settled\",\"timestamp\":\"2026-10-01 21:58:15,634+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-7f6f4599-326\",\"order_id\":\"59b6876b-6ffa-439f-ba3b-2c49651a8a6c\",\"xray_trace_id\":\"1-6abed777-873da867e71bdc9acd6dc784\"}\n"
      },
      {
        "@timestamp": "2026-10-01 21:58:15.631",
        "@message": "{\"level\":\"INFO\",\"location\":\"process:125\",\"message\":\"order already settled\",\"timestamp\":\"2026-10-01 21:58:15,631+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-7f6f4599-325\",\"order_id\":\"fa6b7b56-022e-4f47-ad36-cbe3a9bcd501\",\"xray_trace_id\":\"1-6abed777-873da867e71bdc9acd6dc784\"}\n"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 322915,
    "scan_budget_left_bytes": 20641985
  }
}
  6. Step 6+160 snote_hypotheses
    hypotheses=likely: poison_message causing messages to go to the DLQ after failing max receive count ruled_out: no errors in fulfillment logs (need to check if logs were empty or filtered incorrectly)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  7. Step 7+163 squery_logsfound nothing
    service=fulfillment, query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" | sort @timestamp desc | limit 20, minutes=15
    Result (145 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 158915,
    "scan_budget_left_bytes": 20483070
  }
}
  8. Step 8+163 slist_recent_deploymentsfound nothing
    service=fulfillment, minutes=30
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  9. Step 9+163 sget_function_config
    service=fulfillment
    Result (672 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-fulfillment",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "Qu3/XPxCmXaWPXZdVpICalRSNrIXkZ6si8xPuHY/geY=",
    "environment": {
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "fulfillment_service",
      "PAYMENTS_DEGRADED_MODE_PARAMETER": "/nightshift/flags/payments_degraded_mode",
      "PAYMENTS_FUNCTION_NAME": "nightshift-payments",
      "PAYMENT_TIMEOUT_SECONDS": "3.0",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "fulfillment"
    }
  }
}
  10. Step 10+164 sget_metrics
    metric=Errors, namespace=AWS/Lambda, minutes=15, statistic=Sum
    Result (284 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "21:52:00Z",
        0
      ],
      [
        "21:53:00Z",
        0
      ],
      [
        "21:54:00Z",
        0
      ],
      [
        "21:55:00Z",
        0
      ],
      [
        "21:56:00Z",
        0
      ],
      [
        "21:57:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 6
    }
  }
}
  11. Step 11+166 squery_logsfound nothing
    query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" | sort @timestamp desc | limit 20, service=orders, minutes=15
    Result (145 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 314738,
    "scan_budget_left_bytes": 20168332
  }
}
  12. Step 12+167 squery_logs
    query=fields @timestamp, @message | sort @timestamp desc | limit 20, service=orders, minutes=15
    Result (2,073 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 21:58:23.149",
        "@message": "{\"time\":\"2026-10-01T21:58:23.149Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"cf463478-87d1-4023-aad1-cb7bc8eb3b55\",\"metrics\":{\"durationMs\":330.664,\"billedDurationMs\":331,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"349d883139af29a2\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 21:58:23.129",
        "@message": "{\"_aws\":{\"Timestamp\":1790891903129,\"CloudWatchMetrics\":[{\"Namespace\":\"NightShift\",\"Dimensions\":[[\"service\"]],\"Metrics\":[{\"Name\":\"CheckoutsPlaced\",\"Unit\":\"Count\"}]}]},\"service\":\"orders\",\"CheckoutsPlaced\":[1.0]}\n"
      },
      {
        "@timestamp": "2026-10-01 21:58:23.129",
        "@message": "{\"level\":\"INFO\",\"location\":\"handler:356\",\"message\":\"checkout complete\",\"timestamp\":\"2026-10-01 21:58:23,129+0000\",\"service\":\"orders\",\"correlation_id\":\"load-7f6f4599-337\",\"order_id\":\"240161f3-4c6d-4dde-8e55-13fe36611378\",\"replayed\":false,\"xray_trace_id\":\"1-6abed77e-6d48b5fd0cf1d31549aa9cdc\"}\n"
      },
      {
        "@timestamp": "2026-10-01 21:58:22.818",
        "@message": "{\"time\":\"2026-10-01T21:58:22.818Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"cf463478-87d1-4023-aad1-cb7bc8eb3b55\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"26\",\"tracing\":{\"spanId\":\"349d883139af29a2\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 21:58:22.249",
        "@message": "{\"time\":\"2026-10-01T21:58:22.249Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"27fbb14b-8dda-4752-b072-086f85771a49\",\"metrics\":{\"durationMs\":423.22,\"billedDurationMs\":424,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"652690f59d8c7d75\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6ab...[108 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 481940,
    "scan_budget_left_bytes": 19686392
  }
}
  13. Step 13+169 sget_traces
    service=fulfillment
    Result (609 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-fulfillment",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 114,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.961,
      "p99": 2.781,
      "max": 5.373
    },
    "slowest": [
      {
        "id": "1-6abed638-a6ed8eba7b527a4630c3e263",
        "seconds": 5.373
      },
      {
        "id": "1-6abed639-c97a74b990bb1dd4ca480125",
        "seconds": 3.878
      },
      {
        "id": "1-6abed646-ff9afd6dea0659dc65c1704e",
        "seconds": 2.781
      },
      {
        "id": "1-6abed766-5e11d7ce2c46b63e6831871d",
        "seconds": 1.917
      },
      {
        "id": "1-6abed740-f93dc23bcca6451107727ace",
        "seconds": 1.869
      }
    ]
  }
}
  14. Step 14+169 sget_metricscited as evidence
    dimensions=QueueName=nightshift-placed-orders-dlq, minutes=15, statistic=Maximum, metric=ApproximateNumberOfMessagesVisible, namespace=AWS/SQS
    Result (522 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/SQS ApproximateNumberOfMessagesVisible {'QueueName': 'nightshift-placed-orders-dlq'} Maximum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "21:43:00Z",
        0
      ],
      [
        "21:44:00Z",
        0
      ],
      [
        "21:45:00Z",
        0
      ],
      [
        "21:46:00Z",
        0
      ],
      [
        "21:47:00Z",
        0
      ],
      [
        "21:48:00Z",
        0
      ],
      [
        "21:49:00Z",
        0
      ],
      [
        "21:50:00Z",
        0
      ],
      [
        "21:51:00Z",
        0
      ],
      [
        "21:52:00Z",
        0
      ],
      [
        "21:53:00Z",
        0
      ],
      [
        "21:54:00Z",
        0
      ],
      [
        "21:55:00Z",
        0
      ],
      [
        "21:56:00Z",
        28
      ],
      [
        "21:57:00Z",
        41
      ]
    ],
    "summary": {
      "min": 0,
      "max": 41,
      "count": 15
    }
  }
}
  15. Step 15+169 sfinish_investigationanswer rejected, sent back
    proposed_actions=["Inspect messages in the dead-letter queue (nightshift-placed-orders-dlq) to identify the malformed or failing payload, fix the consumer/fulfillment logic if needed, and redrive valid messages."], summary=The nightshift-dlq-depth alarm fired because 131 messages accumulated in the placed-orders dead-letter queue. This indicates that messages are failing processing repeatedly and being routed to the DLQ after exceeding the maximum receive count., fault_category=poison_message, hypotheses=likely: poison_message causing messages to go to the DLQ after failing max receive count ruled_out: no errors in fulfillment logs (need to check if logs were empty or filtered incorrectly), confidence=90, root_cause_component=placed-orders, actions=[], evidence_steps=2
    Result (141 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "'proposed_actions' must be string, got list",
    "'actions' must be string, got list"
  ]
}
  16. Step 16+170 sfinish_investigation
    actions=, proposed_actions=Inspect messages in the dead-letter queue (nightshift-placed-orders-dlq) to identify the malformed or failing payload, fix the consumer/fulfillment logic if needed, and redrive valid messages., evidence_steps=2,14, hypotheses=likely: poison_message causing messages to go to the DLQ after failing max receive count ruled_out: no errors in fulfillment logs (need to check if logs were empty or filtered incorrectly), fault_category=poison_message, confidence=90, root_cause_component=placed-orders, summary=The nightshift-dlq-depth alarm fired because 131 messages accumulated in the placed-orders dead-letter queue. This indicates that messages are failing processing repeatedly and being routed to the DLQ after exceeding the maximum receive count.
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-dlq-depth

Investigation 732127fac69a, gemini gemini-3.5-flash-lite, started 2026-10-01T21:58:16+00:00.

Answer

Root cause componentplaced-orders
Fault categorypoison_message
Confidence90
Ended becausefinished

Summary

The nightshift-dlq-depth alarm fired because 131 messages accumulated in the placed-orders dead-letter queue. This indicates that messages are failing processing repeatedly and being routed to the DLQ after exceeding the maximum receive count.

Impact

The page: nightshift-dlq-depth was ALARM (since 2026-10-01T21:58:03Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [28.0 (01/10/26 21:52:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T21:58:17+00:00 step 1: get_alarm
  • 2026-10-01T21:58:17+00:00 step 2: get_queue_stats
  • 2026-10-01T21:58:19+00:00 step 3: query_logs service=fulfillment, minutes=15, query=fields @timestamp, @message | filter level = "ERROR... (found nothing)
  • 2026-10-01T21:58:20+00:00 step 4: query_messages_from_dlq_or_logs service=fulfillment, minutes=15, query=fields @timestamp, @message | sort @timestamp desc ... (failed)
  • 2026-10-01T21:58:21+00:00 step 5: query_logs minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20, service=f...
  • 2026-10-01T21:58:21+00:00 step 6: note_hypotheses hypotheses=likely: poison_message causing messages to go to the DLQ after failing max rece...
  • 2026-10-01T21:58:24+00:00 step 7: query_logs service=fulfillment, query=fields @timestamp, @message | filter level = "ERROR" or level =... (found nothing)
  • 2026-10-01T21:58:24+00:00 step 8: list_recent_deployments service=fulfillment, minutes=30 (found nothing)
  • 2026-10-01T21:58:24+00:00 step 9: get_function_config service=fulfillment
  • 2026-10-01T21:58:25+00:00 step 10: get_metrics metric=Errors, namespace=AWS/Lambda, minutes=15, statistic=Sum
  • 2026-10-01T21:58:27+00:00 step 11: query_logs query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" | sort @times... (found nothing)
  • 2026-10-01T21:58:28+00:00 step 12: query_logs query=fields @timestamp, @message | sort @timestamp desc | limit 20, service=orders, minut...
  • 2026-10-01T21:58:30+00:00 step 13: get_traces service=fulfillment
  • 2026-10-01T21:58:30+00:00 step 14: get_metrics dimensions=QueueName=nightshift-placed-orders-dlq, minutes=15, statistic=Maximum, metric=A...
  • 2026-10-01T21:58:30+00:00 step 15: finish_investigation proposed_actions=['Inspect messages in the dead-letter queue (nightshift-placed-orders-dlq... (refused: 'proposed_actions' must be string, got list; 'actions' must be string, got list)
  • 2026-10-01T21:58:31+00:00 step 16: finish_investigation actions=, proposed_actions=Inspect messages in the dead-letter queue (nightshift-placed-or...

Root cause evidence

  • Step 2, get_queue_stats: {"queues":{"nightshift-placed-orders":{"queue":{"name":"nightshift-placed-orders","waiting":0,"in_flight":2,"visibility_timeout_seconds":0},"dead_letter_queue":{"name":"nightshift-placed-orders-dlq","waiting":131,"in_flight":0,"visibility_t...(older result, shortened)
  • Step 14, get_metrics: {"metric":"AWS/SQS ApproximateNumberOfMessagesVisible {'QueueName': 'nightshift-placed-orders-dlq'} Maximum per 60s","window_minutes":15,"points":[["21:43:00Z",0.0],["21:44:00Z",0.0],["21:45:00Z",0.0],["21:46:00Z",0.0],["21:47:00Z",0.0],["2...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Inspect messages in the dead-letter queue (nightshift-placed-orders-dlq) to identify the malformed or failing payload, fix the consumer/fulfillment logic if needed, and redrive valid messages.

Hypotheses

  • likely: poison_message causing messages to go to the DLQ after failing max receive count
  • ruled_out: no errors in fulfillment logs (need to check if logs were empty or filtered incorrectly)

What the investigation cost

  • 6 model calls, 24,665 input and 925 output tokens (25,590 total)
  • 16 steps, 16 s of wall clock
  • 1,285,128 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments