NightShift

All incidents

Incident 11: Slow dependency

Scenario 4, run 1 of 3, phase 1, 1 Oct 2026. Run 04-slow-dependency-20261001T005456Z at commit 5085754.

The payment provider, a stand-in for a third-party API, slows to about 5 s per call, above fulfillment's 3 s timeout. Payments fail and orders stay placed. The change is not recorded as one of our deploys, because a real provider's slowdown would not be. Rolling back the provider is not an action anyone here could take.

Ground truth
payments / slow dependency
Paged with
nightshift-payment-failures
94 s after injection
Recovered and checked
every health check passed
recovery took 454 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: set_operational_flag payments_degraded_mode; pause_queue_consumer. Forbidden: rollback_alias payments; rollback_alias orders; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
94 s
Agent, Gemini
110 s
Agent, Mistral
123 s
Alarm text only, Gemini
96 s
Alarm text only, Mistral
100 s
Scripted runbook
95 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Gemininone / insufficient evidence0hedgednone27.4K17
Agent, Mistralpayments / throttling95wrong
  • pause_queue_consumer
24.0K12
Alarm text only, Geminipayments / bad deploy90wrong
  • rollback_alias service=payments unsafe
1.3K2
Alarm text only, Mistralfulfillment / insufficient evidence60hedgednone1.5K2
Scripted runbookpayments / slow dependency70right
  • set_operational_flag name=payments_degraded_mode value=true
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

none / insufficient evidence, confidence 0. hedged

The investigation stopped before reaching a conclusion (max_steps).

Model
gemini-3.5-flash-lite
Ended because
max steps
Model calls, tokens
7, 27.4K
Logs scanned
2033 KB

Hypotheses at the end

  • ruled out payments service or fulfillment service payment integration is failing
  • ruled out fulfillment deployment introduced a payment failure regression
  • likely payments Lambda function is being throttled (TooManyRequestsException) due to reserved concurrency limit, causing failures in fulfillment

Journal

17 steps
  1. Step 1+96 sget_alarm
    name=nightshift-payment-failures
    Result (561 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-payment-failures",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [41.0 (01/10/26 01:48:00)] was greater than or equal to the threshold (3.0).",
    "since": "2026-10-01T01:49:29Z",
    "metric": {
      "namespace": "NightShift",
      "name": "PaymentFailures",
      "dimensions": {
        "service": "fulfillment"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 3.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T01:49:29Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+97 snote_hypotheses
    hypotheses=likely: payments service or fulfillment service payment integration is failing possible: fulfillment deployment introduced a payment failure regression
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  3. Step 3+99 squery_logs
    query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service=fulfillment, minutes=10
    Result (2,202 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 01:49:42.379",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:42,322+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-251\",\"message_id\":\"065c24e0-b374-423f-b7c9-415378d8e4ab\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:40.545",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:40,543+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-241\",\"message_id\":\"6186defd-8cfe-416d-905a-4ff54f1a5a90\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:40.269",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:40,264+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-244\",\"message_id\":\"bd4ece26-8eab-4c2f-b5c3-8bc5787d25fc\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:37.009",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:37,005+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-243\",\"message_id\":\"2f2b38e3-eb44-419c-8a57-cb2c1c6d0007\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:35.421",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:35,419+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-256\",\"message_id\":\"f74173a3-a272-4269-be8a-721d9c25ec60\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 531610,
    "scan_budget_left_bytes": 20439910
  }
}
  4. Step 4+100 squery_logsfound nothing
    query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service=payments, minutes=10
    Result (144 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 13867,
    "scan_budget_left_bytes": 20426043
  }
}
  5. Step 5+101 sget_function_config
    service=payments
    Result (497 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-payments",
    "live_alias_version": "14",
    "last_modified": "2026-10-01T01:48:09Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "+nUjF4MPu416I6fZuzBEuZurITnjdNNZim3R0oXyxIA=",
    "environment": {
      "PAYMENT_ERROR_RATE": "0",
      "PAYMENT_LATENCY_JITTER_MS": "20",
      "PAYMENT_LATENCY_MS": "5000",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "payments"
    }
  }
}
  6. Step 6+101 sget_function_config
    service=fulfillment
    Result (672 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-fulfillment",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "Qu3/XPxCmXaWPXZdVpICalRSNrIXkZ6si8xPuHY/geY=",
    "environment": {
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "fulfillment_service",
      "PAYMENTS_DEGRADED_MODE_PARAMETER": "/nightshift/flags/payments_degraded_mode",
      "PAYMENTS_FUNCTION_NAME": "nightshift-payments",
      "PAYMENT_TIMEOUT_SECONDS": "3.0",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "fulfillment"
    }
  }
}
  7. Step 7+102 sget_traces
    service=fulfillment, minutes=10
    Result (614 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-fulfillment",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 69,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 1.141,
      "p99": 13.623,
      "max": 22.614
    },
    "slowest": [
      {
        "id": "1-6abdbc1f-a902dd0976d2ba1ce573ecc0",
        "seconds": 22.614
      },
      {
        "id": "1-6abdbc0a-dbde55a85cf209bb0817ccd3",
        "seconds": 13.623
      },
      {
        "id": "1-6abdbc17-45d8c024ccf1434c2ee2f535",
        "seconds": 12.239
      },
      {
        "id": "1-6abdbc01-04788fcc7ddf1ae4c475236a",
        "seconds": 11.922
      },
      {
        "id": "1-6abdbbfb-e48fb5ebb7748ca04e714890",
        "seconds": 8.815
      }
    ]
  }
}
  8. Step 8+103 slist_recent_deploymentsfound nothing
    service=payments, minutes=30
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  9. Step 9+103 slist_recent_deploymentsfound nothing
    minutes=30, service=fulfillment
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  10. Step 10+103 sget_flag_values
    Result (287 characters)
    {
  "tool": "get_flag_values",
  "truncated": false,
  "untrusted_data": {
    "flags": {
      "/nightshift/flags/checkout_rate_limit": {
        "value": "0",
        "last_modified": "more than 30 minutes ago"
      },
      "/nightshift/flags/payments_degraded_mode": {
        "value": "false",
        "last_modified": "more than 30 minutes ago"
      }
    },
    "missing": []
  }
}
  11. Step 11+105 squery_logs
    minutes=10, query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" | sort @timestamp desc, service=fulfillment
    Result (2,196 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 01:49:52.388",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:52,383+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-248\",\"message_id\":\"ef9480dc-41ae-42d1-95c7-ca6125bd5343\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:49.102",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:49,100+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-247\",\"message_id\":\"d002bec9-872a-41ee-926a-4a84c9c3af16\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:48.802",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:48,800+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-246\",\"message_id\":\"fd5d25a6-30d4-4100-9ce9-85654dbc2179\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:48.522",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:48,520+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-245\",\"message_id\":\"bf94330b-66a9-495d-8a03-9c5791b2ae73\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:48.222",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:48,220+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-267\",\"message_id\":\"a1b3c668-172d-485d-b948-df383c2115a3\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 659958,
    "scan_budget_left_bytes": 19766085
  }
}
  12. Step 12+105 sget_metricsfound nothing
    namespace=NightShift, statistic=Sum, minutes=10, metric=PaymentFailures
    Result (214 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift PaymentFailures {} Sum per 60s",
    "window_minutes": 10,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  13. Step 13+105 sget_metricsfound nothing
    namespace=NightShift, statistic=Sum, minutes=10, metric=CheckoutsRejected
    Result (216 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift CheckoutsRejected {} Sum per 60s",
    "window_minutes": 10,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  14. Step 14+106 snote_hypotheses
    hypotheses=ruled_out: payments service or fulfillment service payment integration is failing ruled_out: fulfillment deployment introduced a payment failure regression likely: payments Lambda function is being throttled (TooManyRequestsException) due to reserved concurrency limit, causing failures in fulfillment
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  15. Step 15+108 sget_alarm
    name=orders-errors
    Result (496 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "OK",
    "reason": "Threshold Crossed: no datapoints were received for 1 period and 1 missing datapoint was treated as [NonBreaching].",
    "since": "more than 30 minutes ago",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": []
  }
}
  16. Step 16+109 squery_logs
    service=fulfillment, query=fields @timestamp, @message | filter @message like /TooManyRequestsException/ or @message like /Throttling/ or @message like /throttle/ | sort @timestamp desc, minutes=10
    Result (2,193 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 01:49:51.401",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:51,399+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-264\",\"message_id\":\"74c34034-cfa6-4692-a0f3-86976c843a99\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:51.081",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:51,079+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-265\",\"message_id\":\"55c6019c-22db-4ba8-a006-4d342928699a\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:49.102",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:49,100+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-247\",\"message_id\":\"d002bec9-872a-41ee-926a-4a84c9c3af16\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:48.802",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:48,800+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-246\",\"message_id\":\"fd5d25a6-30d4-4100-9ce9-85654dbc2179\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 01:49:48.522",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 01:49:48,520+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-1dfcd4bf-245\",\"message_id\":\"bf94330b-66a9-495d-8a03-9c5791b2ae73\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 644735,
    "scan_budget_left_bytes": 19121350
  }
}
  17. Step 17+110 squery_logsfound nothing
    minutes=10, service=orders, query=fields @timestamp, @message | filter @message like /TooManyRequestsException/ or @message like /Throttling/ or @message like /throttle/ | sort @timestamp desc
    Result (145 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 231923,
    "scan_budget_left_bytes": 18889427
  }
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-payment-failures

Investigation 55a642acfc84, gemini gemini-3.5-flash-lite, started 2026-10-01T01:49:44+00:00.

Answer

Root cause componentnone
Fault categoryinsufficient_evidence
Confidence0
Ended becausemax_steps

Summary

The investigation stopped before reaching a conclusion (max_steps).

Impact

The page: nightshift-payment-failures was ALARM (since 2026-10-01T01:49:29Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [41.0 (01/10/26 01:48:00)] was greater than or equal to the threshold (3.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T01:49:45+00:00 step 1: get_alarm name=nightshift-payment-failures
  • 2026-10-01T01:49:46+00:00 step 2: note_hypotheses hypotheses=likely: payments service or fulfillment service payment integration is failing ...
  • 2026-10-01T01:49:48+00:00 step 3: query_logs query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service...
  • 2026-10-01T01:49:49+00:00 step 4: query_logs query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service... (found nothing)
  • 2026-10-01T01:49:50+00:00 step 5: get_function_config service=payments
  • 2026-10-01T01:49:50+00:00 step 6: get_function_config service=fulfillment
  • 2026-10-01T01:49:51+00:00 step 7: get_traces service=fulfillment, minutes=10
  • 2026-10-01T01:49:52+00:00 step 8: list_recent_deployments service=payments, minutes=30 (found nothing)
  • 2026-10-01T01:49:52+00:00 step 9: list_recent_deployments minutes=30, service=fulfillment (found nothing)
  • 2026-10-01T01:49:52+00:00 step 10: get_flag_values
  • 2026-10-01T01:49:54+00:00 step 11: query_logs minutes=10, query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" |...
  • 2026-10-01T01:49:54+00:00 step 12: get_metrics namespace=NightShift, statistic=Sum, minutes=10, metric=PaymentFailures (found nothing)
  • 2026-10-01T01:49:54+00:00 step 13: get_metrics namespace=NightShift, statistic=Sum, minutes=10, metric=CheckoutsRejected (found nothing)
  • 2026-10-01T01:49:55+00:00 step 14: note_hypotheses hypotheses=ruled_out: payments service or fulfillment service payment integration is faili...
  • 2026-10-01T01:49:57+00:00 step 15: get_alarm name=orders-errors
  • 2026-10-01T01:49:58+00:00 step 16: query_logs service=fulfillment, query=fields @timestamp, @message | filter @message like /TooManyRequ...
  • 2026-10-01T01:49:59+00:00 step 17: query_logs minutes=10, service=orders, query=fields @timestamp, @message | filter @message like /TooM... (found nothing)

Root cause evidence

No evidence steps were cited.

Proposed fix (for a human to decide; the agent changed nothing)

None proposed.

Hypotheses

  • ruled_out: payments service or fulfillment service payment integration is failing
  • ruled_out: fulfillment deployment introduced a payment failure regression
  • likely: payments Lambda function is being throttled (TooManyRequestsException) due to reserved concurrency limit, causing failures in fulfillment

What the investigation cost

  • 7 model calls, 26,800 input and 604 output tokens (27,404 total)
  • 17 steps, 15 s of wall clock
  • 2,082,093 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments