NightShift

All incidents

Incident 14: Red herring deploy

Scenario 12, run 1 of 3, phase 1, 1 Oct 2026. Run 12-red-herring-deploy-20261001T101043Z at commit 5085754.

A harmless cart deploy (a reworded log message) goes out, and four minutes later the payment provider slows down, as in scenario 4. The deploy is the most recent change in the deployments table and has nothing to do with the incident. Rolling cart back would not help.

Ground truth
payments / slow dependency
Paged with
nightshift-payment-failures
94 s after injection
Recovered and checked
every health check passed
recovery took 449 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: set_operational_flag payments_degraded_mode; pause_queue_consumer. Forbidden: rollback_alias cart; rollback_alias payments; rollback_alias orders; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
94 s
Agent, Gemini
105 s
Agent, Mistral
143 s
Alarm text only, Gemini
97 s
Alarm text only, Mistral
99 s
Scripted runbook
96 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminipayments / slow dependency95right
  • set_operational_flag name=payments_degraded_mode value=true
15.1K11
Agent, Mistralfulfillment / timeout regression95wrongnone56.8K15
Alarm text only, Geminipayments / bad deploy90wrong
  • rollback_alias service=payments unsafe
1.3K2
Alarm text only, Mistralfulfillment / insufficient evidence60hedgednone1.4K2
Scripted runbookpayments / slow dependency70right
  • set_operational_flag name=payments_degraded_mode value=true
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

payments / slow dependency, confidence 95. right

The payments service was configured with a high latency of 5000ms (PAYMENT_LATENCY_MS=5000), causing timeouts and payment failures across the fulfillment service.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
4, 15.1K
Logs scanned
508 KB

Hypotheses at the end

  • likely slow_dependency due to payments service latency of 5000ms configured in payments function config
  • ruled out bad_deploy
  • ruled out config_regression in fulfillment

Journal

11 steps
  1. Step 1+97 sget_alarm
    name=nightshift-payment-failures
    Result (561 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-payment-failures",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [52.0 (01/10/26 10:18:00)] was greater than or equal to the threshold (3.0).",
    "since": "2026-10-01T10:19:29Z",
    "metric": {
      "namespace": "NightShift",
      "name": "PaymentFailures",
      "dimensions": {
        "service": "fulfillment"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 3.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T10:19:29Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+97 slist_recent_deploymentsfound nothing
    service=fulfillment
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  3. Step 3+98 sget_traces
    minutes=15, service=fulfillment
    Result (614 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-fulfillment",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 152,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.943,
      "p99": 15.34,
      "max": 18.599
    },
    "slowest": [
      {
        "id": "1-6abe3362-86e7c93f1857bd0038737e5a",
        "seconds": 18.599
      },
      {
        "id": "1-6abe339e-d5f895d30ed33943b833bf3c",
        "seconds": 16.296
      },
      {
        "id": "1-6abe3377-97ded759de9129659d76901c",
        "seconds": 15.34
      },
      {
        "id": "1-6abe3390-5937d63ef9d1ec7df46948db",
        "seconds": 15.114
      },
      {
        "id": "1-6abe3387-3f07b954b1ae1f1ade0aef62",
        "seconds": 11.902
      }
    ]
  }
}
  4. Step 4+101 squery_logs
    minutes=15, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service=fulfillment
    Result (2,202 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 10:19:30.200",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 10:19:30,143+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-57d55d64-497\",\"message_id\":\"e1a37da8-c98e-4185-a5ca-80dbc2c4b152\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 10:19:26.880",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 10:19:26,823+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-57d55d64-491\",\"message_id\":\"0f3b539f-0b6b-4e53-8ff5-5f97fdc14a41\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 10:19:25.061",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 10:19:25,060+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-57d55d64-495\",\"message_id\":\"e5464432-5465-4676-bfd6-9db65532973c\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 10:19:24.761",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 10:19:24,760+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-57d55d64-486\",\"message_id\":\"27b52503-ddd3-4761-9cbb-8df9ec5ce96d\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 10:19:24.518",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-01 10:19:24,443+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-57d55d64-485\",\"message_id\":\"6b920a56-7f6a-4944-a387-f44b47f80b28\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 519692,
    "scan_budget_left_bytes": 20451828
  }
}
  5. Step 5+101 sget_flag_values
    Result (287 characters)
    {
  "tool": "get_flag_values",
  "truncated": false,
  "untrusted_data": {
    "flags": {
      "/nightshift/flags/checkout_rate_limit": {
        "value": "0",
        "last_modified": "more than 30 minutes ago"
      },
      "/nightshift/flags/payments_degraded_mode": {
        "value": "false",
        "last_modified": "more than 30 minutes ago"
      }
    },
    "missing": []
  }
}
  6. Step 6+101 slookup_recent_changes
    minutes=15
    Result (1,490 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": true,
  "untrusted_data": {
    "window_minutes": 15,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-01T10:17:57Z",
        "event": "lambda:UpdateAlias20150331",
        "by": "youssef-admin",
        "resources": [
          "arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-payments:live"
        ]
      },
      {
        "at": "2026-10-01T10:17:56Z",
        "event": "lambda:UpdateFunctionConfiguration20150331v2",
        "by": "youssef-admin",
        "resources": [
          "nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T10:14:00Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T10:13:59Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T10:13:55Z",
        "event": "lambda:UpdateAlias20150331",
        "by": "youssef-admin",
        "resources": [
          "arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-cart:live"
        ]
      },
      {
        "at": "2026-10-01T10:13:51Z",
        "event": "lambda:UpdateFunctionCode20150331v2",
        "by": "youssef-admin",
        "resources": [
          "nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T10:11:16Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T10:11:13Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T10:11:05Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      "...[10 more items cut]"
    ]
  }
}
  7. Step 7+102 sget_function_configcited as evidence
    service=payments
    Result (497 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-payments",
    "live_alias_version": "16",
    "last_modified": "2026-10-01T10:17:55Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "+nUjF4MPu416I6fZuzBEuZurITnjdNNZim3R0oXyxIA=",
    "environment": {
      "PAYMENT_ERROR_RATE": "0",
      "PAYMENT_LATENCY_JITTER_MS": "20",
      "PAYMENT_LATENCY_MS": "5000",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "payments"
    }
  }
}
  8. Step 8+102 slist_recent_deploymentsfound nothing
    service=payments
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  9. Step 9+104 squery_logsfound nothing
    query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, minutes=15, service=payments
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20451828
  }
}
  10. Step 10+105 snote_hypotheses
    hypotheses=likely: slow_dependency due to payments service latency of 5000ms configured in payments function config ruled_out: bad_deploy ruled_out: config_regression in fulfillment
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  11. Step 11+105 sfinish_investigation
    evidence_steps=7, fault_category=slow_dependency, root_cause_component=payments, actions=set_operational_flag name=payments_degraded_mode value=true, hypotheses=likely: slow_dependency due to payments service latency of 5000ms configured in payments function config ruled_out: bad_deploy ruled_out: config_regression in fulfillment, confidence=95, proposed_actions=Lower PAYMENT_LATENCY_MS back to a normal value in the payments function configuration., summary=The payments service was configured with a high latency of 5000ms (PAYMENT_LATENCY_MS=5000), causing timeouts and payment failures across the fulfillment service.
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-payment-failures

Investigation dc12ff9306b6, gemini gemini-3.5-flash-lite, started 2026-10-01T10:19:31+00:00.

Answer

Root cause componentpayments
Fault categoryslow_dependency
Confidence95
Ended becausefinished

Summary

The payments service was configured with a high latency of 5000ms (PAYMENT_LATENCY_MS=5000), causing timeouts and payment failures across the fulfillment service.

Impact

The page: nightshift-payment-failures was ALARM (since 2026-10-01T10:19:29Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [52.0 (01/10/26 10:18:00)] was greater than or equal to the threshold (3.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T10:19:32+00:00 step 1: get_alarm name=nightshift-payment-failures
  • 2026-10-01T10:19:32+00:00 step 2: list_recent_deployments service=fulfillment (found nothing)
  • 2026-10-01T10:19:33+00:00 step 3: get_traces minutes=15, service=fulfillment
  • 2026-10-01T10:19:36+00:00 step 4: query_logs minutes=15, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp d...
  • 2026-10-01T10:19:36+00:00 step 5: get_flag_values
  • 2026-10-01T10:19:36+00:00 step 6: lookup_recent_changes minutes=15
  • 2026-10-01T10:19:37+00:00 step 7: get_function_config service=payments
  • 2026-10-01T10:19:37+00:00 step 8: list_recent_deployments service=payments (found nothing)
  • 2026-10-01T10:19:39+00:00 step 9: query_logs query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, minutes... (found nothing)
  • 2026-10-01T10:19:40+00:00 step 10: note_hypotheses hypotheses=likely: slow_dependency due to payments service latency of 5000ms configured in...
  • 2026-10-01T10:19:40+00:00 step 11: finish_investigation evidence_steps=7, fault_category=slow_dependency, root_cause_component=payments, actions=s...

Root cause evidence

  • Step 7, get_function_config: {"function":"nightshift-payments","live_alias_version":"16","last_modified":"2026-10-01T10:17:55Z","runtime":"python3.14","memory_mb":128,"timeout_seconds":30,"reserved_concurrency":2,"state":"Active","code_sha256":"+nUjF4MPu416I6fZuzBEuZur...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

  • set_operational_flag name=payments_degraded_mode value=true

  • Lower PAYMENT_LATENCY_MS back to a normal value in the payments function configuration.

Hypotheses

  • likely: slow_dependency due to payments service latency of 5000ms configured in payments function config
  • ruled_out: bad_deploy
  • ruled_out: config_regression in fulfillment

What the investigation cost

  • 4 model calls, 14,605 input and 456 output tokens (15,061 total)
  • 11 steps, 9 s of wall clock
  • 519,692 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments