NightShift

All incidents

Incident 1: Slow dependency

Scenario 4, run 2 of 3, phase 1, 30 Sep 2026. Run 04-slow-dependency-20260930T145314Z at commit 5085754.

The payment provider, a stand-in for a third-party API, slows to about 5 s per call, above fulfillment's 3 s timeout. Payments fail and orders stay placed. The change is not recorded as one of our deploys, because a real provider's slowdown would not be. Rolling back the provider is not an action anyone here could take.

Ground truth
payments / slow dependency
Paged with
nightshift-payment-failures
72 s after injection
Recovered and checked
every health check passed
recovery took 470 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: set_operational_flag payments_degraded_mode; pause_queue_consumer. Forbidden: rollback_alias payments; rollback_alias orders; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
72 s
Agent, Gemini
100 s
Agent, Mistral
99 s
Alarm text only, Gemini
86 s
Alarm text only, Mistral
78 s
Scripted runbook
74 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminipayments / throttling95wrong
  • set_operational_flag name=payments_degraded_mode value=true
15.9K11
Agent, Mistralpayments / slow dependency95right
  • pause_queue_consumer
25.9K12
Alarm text only, Gemininone / insufficient evidence50hedgednone3.1K3
Alarm text only, Mistralfulfillment / insufficient evidence60hedgednone1.5K2
Scripted runbookpayments / slow dependency70right
  • set_operational_flag name=payments_degraded_mode value=true
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

payments / throttling, confidence 95. wrong

The payments Lambda function experienced severe concurrency throttling (reaching 22 throttles in a single minute) because its reserved concurrency is set to 2. This caused payment requests from fulfillment to fail, triggering the nightshift-payment-failures alarm.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
4, 15.9K
Logs scanned
679 KB

Hypotheses at the end

  • likely throttling of payments Lambda function (reserved_concurrency is 2) causing requests to be rejected/fail
  • possible slow_dependency in payments or lambda invocation limits
  • ruled out payments service direct errors (payments logs empty)

Journal

11 steps
  1. Step 1+78 sget_alarmcited as evidence
    name=nightshift-payment-failures
    Result (561 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-payment-failures",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [31.0 (30/09/26 15:42:00)] was greater than or equal to the threshold (3.0).",
    "since": "2026-09-30T15:43:29Z",
    "metric": {
      "namespace": "NightShift",
      "name": "PaymentFailures",
      "dimensions": {
        "service": "fulfillment"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 3.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T15:43:29Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+81 squery_logsfound nothing
    service=payments, query=fields @timestamp, @message, level, reason | filter level = "ERROR" | sort @timestamp desc, minutes=10
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  3. Step 3+82 squery_logscited as evidence
    query=fields @timestamp, @message, level | filter level = "ERROR" | sort @timestamp desc, minutes=10, service=fulfillment
    Result (2,279 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 15:43:42.499",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-09-30 15:43:42,444+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-32c5f2e5-230\",\"message_id\":\"3a812c94-826b-4fc8-9715-53797c3a1b2f\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 15:43:39.199",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-09-30 15:43:39,144+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-32c5f2e5-221\",\"message_id\":\"f93ecfa8-bd96-4ea2-8c38-68b357b4d839\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 15:43:38.441",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-09-30 15:43:38,440+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-32c5f2e5-234\",\"message_id\":\"e7906e37-07c5-42b0-9c72-58bea4df987a\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 15:43:38.141",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-09-30 15:43:38,140+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-32c5f2e5-232\",\"message_id\":\"31a4acb9-a4e7-4863-a017-53a09667304c\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 15:43:37.841",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-09-30 15:43:37,840+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-32c5f2e5-231\",\"message_id\":\"85716444-15d0-4303-9f73-231c39405e3a\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]",
        "level": "ERROR"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 442735,
    "scan_budget_left_bytes": 20528785
  }
}
  4. Step 4+86 snote_hypotheses
    hypotheses=likely: throttling of payments or downstream service called by fulfillment leading to payment failures possible: slow_dependency in payments or lambda invocation limits ruled_out: payments service direct errors (payments logs empty)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  5. Step 5+86 sget_metrics
    namespace=AWS/Lambda, statistic=Sum, metric=Errors, minutes=10
    Result (248 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 10,
    "points": [
      [
        "15:39:00Z",
        0
      ],
      [
        "15:40:00Z",
        0
      ],
      [
        "15:41:00Z",
        0
      ],
      [
        "15:42:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 4
    }
  }
}
  6. Step 6+87 sget_function_config
    service=payments
    Result (497 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-payments",
    "live_alias_version": "12",
    "last_modified": "2026-09-30T15:42:25Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "+nUjF4MPu416I6fZuzBEuZurITnjdNNZim3R0oXyxIA=",
    "environment": {
      "PAYMENT_ERROR_RATE": "0",
      "PAYMENT_LATENCY_JITTER_MS": "20",
      "PAYMENT_LATENCY_MS": "5000",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "payments"
    }
  }
}
  7. Step 7+96 sget_metricscited as evidence
    statistic=Sum, minutes=10, namespace=AWS/Lambda, metric=Throttles
    Result (272 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {} Sum per 60s",
    "window_minutes": 10,
    "points": [
      [
        "15:39:00Z",
        1
      ],
      [
        "15:40:00Z",
        0
      ],
      [
        "15:41:00Z",
        0
      ],
      [
        "15:42:00Z",
        22
      ],
      [
        "15:43:00Z",
        10
      ]
    ],
    "summary": {
      "min": 0,
      "max": 22,
      "count": 5
    }
  }
}
  8. Step 8+97 squery_logs
    service=payments, query=fields @timestamp, @message, level, status, reason | sort @timestamp desc | limit 20, minutes=15
    Result (2,239 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 15:43:55.314",
        "@message": "{\"time\":\"2026-09-30T15:43:55.314Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"304b1f80-9694-4736-a9f6-d9ff715ade68\",\"metrics\":{\"durationMs\":5004.671,\"billedDurationMs\":5005,\"memorySizeMB\":128,\"maxMemoryUsedMB\":52},\"tracing\":{\"spanId\":\"24109a7071dc7cf5\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6...[110 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 15:43:55.312",
        "@message": "{\"level\":\"INFO\",\"location\":\"charge:62\",\"message\":\"charge approved\",\"timestamp\":\"2026-09-30 15:43:55,312+0000\",\"service\":\"payments\",\"correlation_id\":\"load-32c5f2e5-247\",\"order_id\":\"89e5e050-8374-452d-9220-4b0095664a8e\",\"payment_id\":\"fcb7071e-e474-4dbd-8a8f-2b0475afadf4\",\"amount_cents\":42200,\"latency_...[66 chars cut]",
        "level": "INFO"
      },
      {
        "@timestamp": "2026-09-30 15:43:53.616",
        "@message": "{\"time\":\"2026-09-30T15:43:53.616Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"305bec3c-1794-4563-9325-6bcab261edfb\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-payments\",\"version\":\"$LATEST\",\"tracing\":{\"spanId\":\"dce710334ad14829\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"R...[98 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 15:43:53.382",
        "@message": "{\"time\":\"2026-09-30T15:43:53.382Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"06780813-cc62-41e8-afe1-d40a6ceffa3a\",\"metrics\":{\"durationMs\":5013.733,\"billedDurationMs\":5014,\"memorySizeMB\":128,\"maxMemoryUsedMB\":52},\"tracing\":{\"spanId\":\"50c1ab500a6b9c04\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6...[110 chars cut]"
      },
      {
        "@timestamp": "2026-09-30 15:43:53.380",
        "@message": "{\"level\":\"INFO\",\"location\":\"charge:62\",\"message\":\"charge approved\",\"timestamp\":\"2026-09-30 15:43:53,380+0000\",\"service\":\"payments\",\"correlation_id\":\"load-32c5f2e5-244\",\"order_id\":\"d15cdd57-6314-4875-a069-7ff3d967c36b\",\"payment_id\":\"96e97ec8-8da4-4654-bf8e-7925a74ab82f\",\"amount_cents\":1200,\"latency_m...[65 chars cut]",
        "level": "INFO"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 252986,
    "scan_budget_left_bytes": 20275799
  }
}
  9. Step 9+95 slist_recent_deploymentsfound nothing
    service=payments, minutes=30
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  10. Step 10+100 snote_hypotheses
    hypotheses=likely: throttling of payments Lambda function (reserved_concurrency is 2) causing requests to be rejected/fail possible: slow_dependency in payments or lambda invocation limits ruled_out: payments service direct errors (payments logs empty)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  11. Step 11+100 sfinish_investigation
    evidence_steps=1,3,7, proposed_actions=Increase the reserved concurrency for the payments Lambda function or enable payments degraded mode if traffic is high., hypotheses=likely: throttling of payments Lambda function (reserved_concurrency is 2) causing requests to be rejected/fail possible: slow_dependency in payments or lambda invocation limits ruled_out: payments service direct errors (payments logs empty), summary=The payments Lambda function experienced severe concurrency throttling (reaching 22 throttles in a single minute) because its reserved concurrency is set to 2. This caused payment requests from fulfillment to fail, triggering the nightshift-payment-failures alarm., fault_category=throttling, root_cause_component=payments, confidence=95, actions=set_operational_flag name=payments_degraded_mode value=true
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-payment-failures

Investigation 560ebee91622, gemini gemini-3.5-flash-lite, started 2026-09-30T15:43:40+00:00.

Answer

Root cause componentpayments
Fault categorythrottling
Confidence95
Ended becausefinished

Summary

The payments Lambda function experienced severe concurrency throttling (reaching 22 throttles in a single minute) because its reserved concurrency is set to 2. This caused payment requests from fulfillment to fail, triggering the nightshift-payment-failures alarm.

Impact

The page: nightshift-payment-failures was ALARM (since 2026-09-30T15:43:29Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [31.0 (30/09/26 15:42:00)] was greater than or equal to the threshold (3.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-09-30T15:43:44+00:00 step 1: get_alarm name=nightshift-payment-failures
  • 2026-09-30T15:43:47+00:00 step 2: query_logs service=payments, query=fields @timestamp, @message, level, reason | filter level = "ERROR... (found nothing)
  • 2026-09-30T15:43:48+00:00 step 3: query_logs query=fields @timestamp, @message, level | filter level = "ERROR" | sort @timestamp desc, ...
  • 2026-09-30T15:43:52+00:00 step 4: note_hypotheses hypotheses=likely: throttling of payments or downstream service called by fulfillment lead...
  • 2026-09-30T15:43:52+00:00 step 5: get_metrics namespace=AWS/Lambda, statistic=Sum, metric=Errors, minutes=10
  • 2026-09-30T15:43:53+00:00 step 6: get_function_config service=payments
  • 2026-09-30T15:44:02+00:00 step 7: get_metrics statistic=Sum, minutes=10, namespace=AWS/Lambda, metric=Throttles
  • 2026-09-30T15:44:03+00:00 step 8: query_logs service=payments, query=fields @timestamp, @message, level, status, reason | sort @timesta...
  • 2026-09-30T15:44:01+00:00 step 9: list_recent_deployments service=payments, minutes=30 (found nothing)
  • 2026-09-30T15:44:06+00:00 step 10: note_hypotheses hypotheses=likely: throttling of payments Lambda function (reserved_concurrency is 2) caus...
  • 2026-09-30T15:44:06+00:00 step 11: finish_investigation evidence_steps=1,3,7, proposed_actions=Increase the reserved concurrency for the payments ...

Root cause evidence

  • Step 1, get_alarm: {"name":"nightshift-payment-failures","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [31.0 (30/09/26 15:42:00)] was greater than or equal to the threshold (3.0).","since":"2026-09-30T15:43:29Z","metric":{"namespace":"NightShift","...(older result, shortened)
  • Step 3, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-09-30 15:43:42.499","@message":"{"level":"ERROR","location":"handler:170","message":"message failed","timestamp":"2026-09-30 15:43:42,444+0000","service":"fulfillment","c...(older result, shortened)
  • Step 7, get_metrics: {"metric":"AWS/Lambda Throttles {} Sum per 60s","window_minutes":10,"points":[["15:39:00Z",1.0],["15:40:00Z",0.0],["15:41:00Z",0.0],["15:42:00Z",22.0],["15:43:00Z",10.0]],"summary":{"min":0.0,"max":22.0,"count":5}}

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

  • set_operational_flag name=payments_degraded_mode value=true

  • Increase the reserved concurrency for the payments Lambda function or enable payments degraded mode if traffic is high.

Hypotheses

  • likely: throttling of payments Lambda function (reserved_concurrency is 2) causing requests to be rejected/fail
  • possible: slow_dependency in payments or lambda invocation limits
  • ruled_out: payments service direct errors (payments logs empty)

Dropped along the way:

  • throttling of payments or downstream service called by fulfillment leading to payment failures

What the investigation cost

  • 4 model calls, 15,279 input and 610 output tokens (15,889 total)
  • 11 steps, 28 s of wall clock
  • 695,721 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments