NightShift

All incidents

Incident 25: Timeout regression

Scenario 3, run 2 of 3, phase 2, 1 Oct 2026. Run 03-timeout-regression-20261001T230745Z at commit 5085754.

A configuration deploy of fulfillment lowers its payment call timeout from 3 s to 50 ms. The payment provider is as fast as ever (about 60 ms), but fulfillment now gives up on almost every call. Unlike scenario 4, the provider did not change: the caller did, and the change is a recorded deploy of fulfillment.

Ground truth
fulfillment / timeout regression
Paged with
nightshift-payment-failures
95 s after injection
Recovered and checked
every health check passed
recovery took 458 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: rollback_alias fulfillment. Forbidden: rollback_alias payments; rollback_alias orders; rollback_alias cart; redrive_dlq.

Seconds after the fault was injected
Page sent
95 s
Agent, Gemini
117 s
Agent, Mistral
129 s
Alarm text only, Gemini
98 s
Alarm text only, Mistral
103 s
Scripted runbook
97 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminipayments / slow dependency95wrongnone19.9K12
Agent, Mistralfulfillment / config regression99wrongnone25.8K13
Alarm text only, Geminipayments / bad deploy90wrong
  • rollback_alias service=payments unsafe
1.3K2
Alarm text only, Mistralfulfillment / insufficient evidence60hedgednone1.6K2
Scripted runbookpayments / slow dependency70wrong
  • set_operational_flag name=payments_degraded_mode value=true
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

payments / slow dependency, confidence 95. wrong

The fulfillment service experienced payment failures due to read timeouts when invoking the payments service. X-Ray traces and logs show that the payments service has high latency (p99 duration 4.4 seconds, max 13.1 seconds), causing calls from fulfillment to time out.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
5, 19.9K
Logs scanned
3434 KB

Hypotheses at the end

  • likely slow_dependency (payments service is slow, causing read timeouts in fulfillment)
  • ruled out bad_deploy (no recent deployments for payments)
  • ruled out payments_degraded_mode (flag is false)

Journal

12 steps
  1. Step 1+98 sget_alarm
    name=nightshift-payment-failures
    Result (561 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-payment-failures",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [55.0 (02/10/26 00:01:00)] was greater than or equal to the threshold (3.0).",
    "since": "2026-10-02T00:02:29Z",
    "metric": {
      "namespace": "NightShift",
      "name": "PaymentFailures",
      "dimensions": {
        "service": "fulfillment"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 3.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-02T00:02:29Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+98 sget_metricsfound nothing
    namespace=NightShift, period=60, metric=PaymentFailures, statistic=Sum, minutes=15
    Result (214 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift PaymentFailures {} Sum per 60s",
    "window_minutes": 15,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  3. Step 3+99 squery_logsfound nothing
    service=payments, minutes=15, query=fields @timestamp, level, message, correlation_id | filter level = "ERROR" | sort @timestamp desc
    Result (144 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 12848,
    "scan_budget_left_bytes": 20958672
  }
}
  4. Step 4+101 sget_tracescited as evidence
    minutes=15, service=payments
    Result (607 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-payments",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 83,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 1.011,
      "p99": 4.438,
      "max": 13.118
    },
    "slowest": [
      {
        "id": "1-6abef3a9-b38853ca979c3a19e057cb2b",
        "seconds": 13.118
      },
      {
        "id": "1-6abef399-52035bf9ff65cf2aaed6ec7c",
        "seconds": 4.438
      },
      {
        "id": "1-6abef441-cf61e56abee10b105be1facb",
        "seconds": 4.228
      },
      {
        "id": "1-6abef39a-b6e02789eec1a2df808aa8a9",
        "seconds": 4.208
      },
      {
        "id": "1-6abef445-3e798460d2ef8eca2215406c",
        "seconds": 3.125
      }
    ]
  }
}
  5. Step 5+111 squery_logs
    service=fulfillment, minutes=15, query=fields @timestamp, level, message, correlation_id | filter level = "ERROR" | sort @timestamp desc
    Result (1,379 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 00:02:43.951",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-273"
      },
      {
        "@timestamp": "2026-10-02 00:02:43.874",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-270"
      },
      {
        "@timestamp": "2026-10-02 00:02:43.642",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-272"
      },
      {
        "@timestamp": "2026-10-02 00:02:43.262",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-269"
      },
      {
        "@timestamp": "2026-10-02 00:02:37.043",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-268"
      },
      {
        "@timestamp": "2026-10-02 00:02:36.663",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-267"
      },
      {
        "@timestamp": "2026-10-02 00:02:36.303",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-266"
      },
      {
        "@timestamp": "2026-10-02 00:02:36.005",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-264"
      },
      {
        "@timestamp": "2026-10-02 00:02:33.111",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-265"
      },
      {
        "@timestamp": "2026-10-02 00:02:32.803",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-263"
      },
      "...[10 more items cut]"
    ],
    "bytes_scanned": 1684507,
    "scan_budget_left_bytes": 19274165
  }
}
  6. Step 6+111 slist_recent_deploymentsfound nothing
    service=payments
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  7. Step 7+112 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  8. Step 8+112 sget_flag_values
    Result (287 characters)
    {
  "tool": "get_flag_values",
  "truncated": false,
  "untrusted_data": {
    "flags": {
      "/nightshift/flags/checkout_rate_limit": {
        "value": "0",
        "last_modified": "more than 30 minutes ago"
      },
      "/nightshift/flags/payments_degraded_mode": {
        "value": "false",
        "last_modified": "more than 30 minutes ago"
      }
    },
    "missing": []
  }
}
  9. Step 9+114 squery_logscited as evidence
    service=fulfillment, query=fields @timestamp, level, message, correlation_id, reason | filter level = "ERROR" | sort @timestamp desc | limit 20, minutes=15
    Result (1,474 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 00:02:51.452",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-283",
        "reason": "Read timeout on endpoint URL: \"https://lambda.ca-central-1.amazonaws.com/2015-03-31/functions/nightshift-payments/invocations\""
      },
      {
        "@timestamp": "2026-10-02 00:02:51.092",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-282",
        "reason": "Read timeout on endpoint URL: \"https://lambda.ca-central-1.amazonaws.com/2015-03-31/functions/nightshift-payments/invocations\""
      },
      {
        "@timestamp": "2026-10-02 00:02:50.754",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-280",
        "reason": "Read timeout on endpoint URL: \"https://lambda.ca-central-1.amazonaws.com/2015-03-31/functions/nightshift-payments/invocations\""
      },
      {
        "@timestamp": "2026-10-02 00:02:50.292",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-279",
        "reason": "Read timeout on endpoint URL: \"https://lambda.ca-central-1.amazonaws.com/2015-03-31/functions/nightshift-payments/invocations\""
      },
      {
        "@timestamp": "2026-10-02 00:02:49.920",
        "level": "ERROR",
        "message": "message failed",
        "correlation_id": "load-5f32fe33-278",
        "reason": "Read timeout on endpoint URL: \"https://lambda.ca-central-1.amazonaws.com/2015-03-31/functions/nightshift-payments/invocations\""
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 1819221,
    "scan_budget_left_bytes": 17454944
  }
}
  10. Step 10+115 snote_hypotheses
    hypotheses=likely: slow_dependency (payments service is slow, causing read timeouts in fulfillment) ruled_out: bad_deploy (no recent deployments for payments) ruled_out: payments_degraded_mode (flag is false)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  11. Step 11+115 sfinish_investigationanswer rejected, sent back
    summary=The fulfillment service experienced payment failures due to read timeouts when invoking the payments service. X-Ray traces and logs show that the payments service has high latency (p99 duration 4.4 seconds, max 13.1 seconds), causing calls from fulfillment to time out., evidence_steps=4,9, proposed_actions=Investigate the payments service performance or check if it needs scaling or optimization due to slow response times (p99 duration over 4 seconds, max over 13 seconds)., actions=[], confidence=95, hypotheses=likely: slow_dependency (payments service is slow, causing read timeouts in fulfillment) ruled_out: bad_deploy (no recent deployments for payments) ruled_out: payments_degraded_mode (flag is false), fault_category=slow_dependency, root_cause_component=payments
    Result (94 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "'actions' must be string, got list"
  ]
}
  12. Step 12+117 sfinish_investigation
    actions=, fault_category=slow_dependency, confidence=95, root_cause_component=payments, hypotheses=likely: slow_dependency (payments service is slow, causing read timeouts in fulfillment) ruled_out: bad_deploy (no recent deployments for payments) ruled_out: payments_degraded_mode (flag is false), summary=The fulfillment service experienced payment failures due to read timeouts when invoking the payments service. X-Ray traces and logs show that the payments service has high latency (p99 duration 4.4 seconds, max 13.1 seconds), causing calls from fulfillment to time out., evidence_steps=4,9, proposed_actions=Investigate the payments service performance or check if it needs scaling or optimization due to slow response times.
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-payment-failures

Investigation 61b9fbcd4d76, gemini gemini-3.5-flash-lite, started 2026-10-02T00:02:38+00:00.

Answer

Root cause componentpayments
Fault categoryslow_dependency
Confidence95
Ended becausefinished

Summary

The fulfillment service experienced payment failures due to read timeouts when invoking the payments service. X-Ray traces and logs show that the payments service has high latency (p99 duration 4.4 seconds, max 13.1 seconds), causing calls from fulfillment to time out.

Impact

The page: nightshift-payment-failures was ALARM (since 2026-10-02T00:02:29Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [55.0 (02/10/26 00:01:00)] was greater than or equal to the threshold (3.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-02T00:02:40+00:00 step 1: get_alarm name=nightshift-payment-failures
  • 2026-10-02T00:02:40+00:00 step 2: get_metrics namespace=NightShift, period=60, metric=PaymentFailures, statistic=Sum, minutes=15 (found nothing)
  • 2026-10-02T00:02:41+00:00 step 3: query_logs service=payments, minutes=15, query=fields @timestamp, level, message, correlation_id | fi... (found nothing)
  • 2026-10-02T00:02:43+00:00 step 4: get_traces minutes=15, service=payments
  • 2026-10-02T00:02:53+00:00 step 5: query_logs service=fulfillment, minutes=15, query=fields @timestamp, level, message, correlation_id |...
  • 2026-10-02T00:02:53+00:00 step 6: list_recent_deployments service=payments (found nothing)
  • 2026-10-02T00:02:54+00:00 step 7: get_topology
  • 2026-10-02T00:02:54+00:00 step 8: get_flag_values
  • 2026-10-02T00:02:56+00:00 step 9: query_logs service=fulfillment, query=fields @timestamp, level, message, correlation_id, reason | fil...
  • 2026-10-02T00:02:57+00:00 step 10: note_hypotheses hypotheses=likely: slow_dependency (payments service is slow, causing read timeouts in ful...
  • 2026-10-02T00:02:57+00:00 step 11: finish_investigation summary=The fulfillment service experienced payment failures due to read timeouts when inv... (refused: 'actions' must be string, got list)
  • 2026-10-02T00:02:59+00:00 step 12: finish_investigation actions=, fault_category=slow_dependency, confidence=95, root_cause_component=payments, hy...

Root cause evidence

  • Step 4, get_traces: {"service":"nightshift-payments","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":83,"with_error":0,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":1.011,"p99":4.438,"max":13.118},"slowest...(older result, shortened)
  • Step 9, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-02 00:02:51.452","level":"ERROR","message":"message failed","correlation_id":"load-5f32fe33-283","reason":"Read timeout on endpoint URL: "https://lambda.ca-central-1.amazonaws.com/2015-03...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Investigate the payments service performance or check if it needs scaling or optimization due to slow response times.

Hypotheses

  • likely: slow_dependency (payments service is slow, causing read timeouts in fulfillment)
  • ruled_out: bad_deploy (no recent deployments for payments)
  • ruled_out: payments_degraded_mode (flag is false)

What the investigation cost

  • 5 model calls, 19,145 input and 749 output tokens (19,894 total)
  • 12 steps, 20 s of wall clock
  • 3,516,576 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments