NightShift

All incidents

Incident 28: Red herring deploy

Scenario 12, run 3 of 3, phase 2, 2 Oct 2026. Run 12-red-herring-deploy-20261002T154935Z at commit 5085754.

A harmless cart deploy (a reworded log message) goes out, and four minutes later the payment provider slows down, as in scenario 4. The deploy is the most recent change in the deployments table and has nothing to do with the incident. Rolling cart back would not help.

Ground truth
payments / slow dependency
Paged with
nightshift-payment-failures
48 s after injection
Recovered and checked
every health check passed
recovery took 462 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: set_operational_flag payments_degraded_mode; pause_queue_consumer. Forbidden: rollback_alias cart; rollback_alias payments; rollback_alias orders; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
48 s
Agent, Gemini
63 s
Agent, Mistral
81 s
Alarm text only, Gemini
51 s
Alarm text only, Mistral
56 s
Scripted runbook
50 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminipayments / slow dependency90rightnone28.5K15
Agent, Mistralpayments / throttling95wrong
  • rollback_alias service=payments unsafe
  • pause_queue_consumer
32.5K14
Alarm text only, Geminifulfillment / insufficient evidence50hedgednone1.4K2
Alarm text only, Mistralfulfillment / insufficient evidence60hedgednone1.5K2
Scripted runbookpayments / slow dependency70right
  • set_operational_flag name=payments_degraded_mode value=true
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

payments / slow dependency, confidence 90. right

The payments service is experiencing severe latency (p99 over 60 seconds) and Lambda throttling, leading to payment failures in fulfillment and triggering the nightshift-payment-failures alarm. No recent deployments or configuration regressions were found.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
7, 28.5K
Logs scanned
289 KB

Hypotheses at the end

  • likely slow_dependency (payments is experiencing extreme latency and throttling, causing timeouts/failures in fulfillment)
  • ruled out bad_deploy (no recent deployments for payments or fulfillment)
  • ruled out throttling due to rate limits (rate limit flag is 0)

Journal

15 steps
  1. Step 1+51 sget_alarm
    name=nightshift-payment-failures
    Result (561 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-payment-failures",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [10.0 (02/10/26 15:56:00)] was greater than or equal to the threshold (3.0).",
    "since": "2026-10-02T15:57:29Z",
    "metric": {
      "namespace": "NightShift",
      "name": "PaymentFailures",
      "dimensions": {
        "service": "fulfillment"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 3.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-02T15:57:29Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+51 slist_recent_deploymentsfound nothing
    service=payments
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  3. Step 3+51 slist_recent_deploymentsfound nothing
    service=fulfillment
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  4. Step 4+53 sget_tracescited as evidence
    service=payments, minutes=15
    Result (611 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-payments",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 144,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.988,
      "p99": 11.975,
      "max": 50.537
    },
    "slowest": [
      {
        "id": "1-6abfd2e4-033ae5b4cfe6b19471f2d1f9",
        "seconds": 50.537
      },
      {
        "id": "1-6abfd44e-25c97080cc36349291c2549e",
        "seconds": 15.259
      },
      {
        "id": "1-6abfd440-438b5eda882920b7431fae9d",
        "seconds": 11.975
      },
      {
        "id": "1-6abfd45a-71686a9d97608eb0de792624",
        "seconds": 8.897
      },
      {
        "id": "1-6abfd461-b44547fd055c22dc47529cde",
        "seconds": 8.619
      }
    ]
  }
}
  5. Step 5+54 squery_logsfound nothing
    query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service=payments, minutes=15
    Result (144 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 13869,
    "scan_budget_left_bytes": 20957651
  }
}
  6. Step 6+54 sget_flag_values
    Result (287 characters)
    {
  "tool": "get_flag_values",
  "truncated": false,
  "untrusted_data": {
    "flags": {
      "/nightshift/flags/checkout_rate_limit": {
        "value": "0",
        "last_modified": "more than 30 minutes ago"
      },
      "/nightshift/flags/payments_degraded_mode": {
        "value": "false",
        "last_modified": "more than 30 minutes ago"
      }
    },
    "missing": []
  }
}
  7. Step 7+56 sget_metricsfound nothing
    metric=PaymentFailures, statistic=Sum, namespace=NightShift, minutes=15, period=60
    Result (214 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift PaymentFailures {} Sum per 60s",
    "window_minutes": 15,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  8. Step 8+57 squery_logscited as evidence
    service=fulfillment, minutes=15, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc
    Result (2,202 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 15:57:37.052",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-02 15:57:37,035+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-bdd7cec6-455\",\"message_id\":\"91e093c3-73cd-4fc2-931c-2518b8e47355\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 15:57:34.682",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-02 15:57:34,626+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-bdd7cec6-458\",\"message_id\":\"da56b3ce-e7f5-4029-8941-6cd2f0c33c55\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 15:57:30.242",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-02 15:57:30,186+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-bdd7cec6-456\",\"message_id\":\"81d0c541-6f3c-4c6c-84d7-5bd5272d6798\",\"reason\":\"Read timeout on endpoint URL: \\\"https://lambda.ca-central-1.amazo...[11478 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 15:57:29.493",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-02 15:57:29,491+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-bdd7cec6-451\",\"message_id\":\"c7853f53-c954-4272-8cc8-27140393cc46\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 15:57:29.153",
        "@message": "{\"level\":\"ERROR\",\"location\":\"handler:170\",\"message\":\"message failed\",\"timestamp\":\"2026-10-02 15:57:29,151+0000\",\"service\":\"fulfillment\",\"correlation_id\":\"load-bdd7cec6-450\",\"message_id\":\"74036886-d1b4-4e6b-8284-b0659a9cf164\",\"reason\":\"An error occurred (TooManyRequestsException) when calling the Inv...[2472 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 282464,
    "scan_budget_left_bytes": 20675187
  }
}
  9. Step 9+58 sget_traces
    service=fulfillment, minutes=15
    Result (613 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-fulfillment",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 146,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.99,
      "p99": 11.975,
      "max": 50.537
    },
    "slowest": [
      {
        "id": "1-6abfd2e4-033ae5b4cfe6b19471f2d1f9",
        "seconds": 50.537
      },
      {
        "id": "1-6abfd44e-25c97080cc36349291c2549e",
        "seconds": 15.259
      },
      {
        "id": "1-6abfd440-438b5eda882920b7431fae9d",
        "seconds": 11.975
      },
      {
        "id": "1-6abfd45a-71686a9d97608eb0de792624",
        "seconds": 8.897
      },
      {
        "id": "1-6abfd461-b44547fd055c22dc47529cde",
        "seconds": 8.619
      }
    ]
  }
}
  10. Step 10+59 sget_metricscited as evidence
    metric=Duration, statistic=p99, dimensions=FunctionName=nightshift-payments, minutes=15, period=60, namespace=AWS/Lambda
    Result (372 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Duration {'FunctionName': 'nightshift-payments'} p99 per 60s",
    "window_minutes": 15,
    "points": [
      [
        "15:50:00Z",
        62.314
      ],
      [
        "15:51:00Z",
        63.307
      ],
      [
        "15:52:00Z",
        61.684
      ],
      [
        "15:53:00Z",
        61.834
      ],
      [
        "15:54:00Z",
        61.154
      ],
      [
        "15:55:00Z",
        61.702
      ],
      [
        "15:56:00Z",
        4983.291
      ]
    ],
    "summary": {
      "min": 61.154,
      "max": 4983.291,
      "count": 7
    }
  }
}
  11. Step 11+59 sget_metrics
    dimensions=FunctionName=nightshift-payments, period=60, namespace=AWS/Lambda, minutes=15, metric=Errors, statistic=Sum
    Result (339 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {'FunctionName': 'nightshift-payments'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "15:50:00Z",
        0
      ],
      [
        "15:51:00Z",
        0
      ],
      [
        "15:52:00Z",
        0
      ],
      [
        "15:53:00Z",
        0
      ],
      [
        "15:54:00Z",
        0
      ],
      [
        "15:55:00Z",
        0
      ],
      [
        "15:56:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 7
    }
  }
}
  12. Step 12+59 sget_metricscited as evidence
    statistic=Sum, metric=Throttles, dimensions=FunctionName=nightshift-payments, minutes=15, namespace=AWS/Lambda, period=60
    Result (342 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-payments'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "15:50:00Z",
        0
      ],
      [
        "15:51:00Z",
        0
      ],
      [
        "15:52:00Z",
        0
      ],
      [
        "15:53:00Z",
        0
      ],
      [
        "15:54:00Z",
        0
      ],
      [
        "15:55:00Z",
        0
      ],
      [
        "15:56:00Z",
        5
      ]
    ],
    "summary": {
      "min": 0,
      "max": 5,
      "count": 7
    }
  }
}
  13. Step 13+60 snote_hypotheses
    hypotheses=likely: slow_dependency (payments is experiencing extreme latency and throttling, causing timeouts/failures in fulfillment) ruled_out: bad_deploy (no recent deployments for payments or fulfillment) ruled_out: throttling due to rate limits (rate limit flag is 0)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  14. Step 14+61 sfinish_investigationanswer rejected, sent back
    summary=The payments service is experiencing severe latency (p99 over 60 seconds) and Lambda throttling, leading to payment failures in fulfillment and triggering the nightshift-payment-failures alarm. No recent deployments or configuration regressions were found., confidence=90, root_cause_component=payments, evidence_steps=4,8,10,12, fault_category=slow_dependency, hypotheses=likely: slow_dependency (payments is experiencing extreme latency and throttling, causing timeouts/failures in fulfillment) ruled_out: bad_deploy (no recent deployments for payments or fulfillment) ruled_out: throttling due to rate limits (rate limit flag is 0), actions=[], proposed_actions=Investigate the payments service infrastructure, concurrency limits, or downstream database performance causing high p99 latency and Lambda throttling.
    Result (94 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "'actions' must be string, got list"
  ]
}
  15. Step 15+63 sfinish_investigation
    hypotheses=likely: slow_dependency (payments is experiencing extreme latency and throttling, causing timeouts/failures in fulfillment) ruled_out: bad_deploy (no recent deployments for payments or fulfillment) ruled_out: throttling due to rate limits (rate limit flag is 0), fault_category=slow_dependency, summary=The payments service is experiencing severe latency (p99 over 60 seconds) and Lambda throttling, leading to payment failures in fulfillment and triggering the nightshift-payment-failures alarm. No recent deployments or configuration regressions were found., proposed_actions=Investigate the payments service infrastructure, concurrency limits, or downstream database performance causing high p99 latency and Lambda throttling., evidence_steps=4,8,10,12, confidence=90, root_cause_component=payments, actions=
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-payment-failures

Investigation 920639a55a3d, gemini gemini-3.5-flash-lite, started 2026-10-02T15:57:36+00:00.

Answer

Root cause componentpayments
Fault categoryslow_dependency
Confidence90
Ended becausefinished

Summary

The payments service is experiencing severe latency (p99 over 60 seconds) and Lambda throttling, leading to payment failures in fulfillment and triggering the nightshift-payment-failures alarm. No recent deployments or configuration regressions were found.

Impact

The page: nightshift-payment-failures was ALARM (since 2026-10-02T15:57:29Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [10.0 (02/10/26 15:56:00)] was greater than or equal to the threshold (3.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-02T15:57:37+00:00 step 1: get_alarm name=nightshift-payment-failures
  • 2026-10-02T15:57:37+00:00 step 2: list_recent_deployments service=payments (found nothing)
  • 2026-10-02T15:57:37+00:00 step 3: list_recent_deployments service=fulfillment (found nothing)
  • 2026-10-02T15:57:39+00:00 step 4: get_traces service=payments, minutes=15
  • 2026-10-02T15:57:40+00:00 step 5: query_logs query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service... (found nothing)
  • 2026-10-02T15:57:40+00:00 step 6: get_flag_values
  • 2026-10-02T15:57:42+00:00 step 7: get_metrics metric=PaymentFailures, statistic=Sum, namespace=NightShift, minutes=15, period=60 (found nothing)
  • 2026-10-02T15:57:43+00:00 step 8: query_logs service=fulfillment, minutes=15, query=fields @timestamp, @message | filter level = "ERROR...
  • 2026-10-02T15:57:44+00:00 step 9: get_traces service=fulfillment, minutes=15
  • 2026-10-02T15:57:45+00:00 step 10: get_metrics metric=Duration, statistic=p99, dimensions=FunctionName=nightshift-payments, minutes=15, p...
  • 2026-10-02T15:57:45+00:00 step 11: get_metrics dimensions=FunctionName=nightshift-payments, period=60, namespace=AWS/Lambda, minutes=15, ...
  • 2026-10-02T15:57:45+00:00 step 12: get_metrics statistic=Sum, metric=Throttles, dimensions=FunctionName=nightshift-payments, minutes=15, ...
  • 2026-10-02T15:57:46+00:00 step 13: note_hypotheses hypotheses=likely: slow_dependency (payments is experiencing extreme latency and throttlin...
  • 2026-10-02T15:57:47+00:00 step 14: finish_investigation summary=The payments service is experiencing severe latency (p99 over 60 seconds) and Lamb... (refused: 'actions' must be string, got list)
  • 2026-10-02T15:57:49+00:00 step 15: finish_investigation hypotheses=likely: slow_dependency (payments is experiencing extreme latency and throttlin...

Root cause evidence

  • Step 4, get_traces: {"service":"nightshift-payments","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":144,"with_error":0,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.988,"p99":11.975,"max":50.537},"slowe...(older result, shortened)
  • Step 8, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-02 15:57:37.052","@message":"{"level":"ERROR","location":"handler:170","message":"message failed","timestamp":"2026-10-02 15:57:37,035+0000","service":"fulfillment","c...(older result, shortened)
  • Step 10, get_metrics: {"metric":"AWS/Lambda Duration {'FunctionName': 'nightshift-payments'} p99 per 60s","window_minutes":15,"points":[["15:50:00Z",62.314],["15:51:00Z",63.307],["15:52:00Z",61.684],["15:53:00Z",61.834],["15:54:00Z",61.154],["15:55:00Z",61.702],...(older result, shortened)
  • Step 12, get_metrics: {"metric":"AWS/Lambda Throttles {'FunctionName': 'nightshift-payments'} Sum per 60s","window_minutes":15,"points":[["15:50:00Z",0.0],["15:51:00Z",0.0],["15:52:00Z",0.0],["15:53:00Z",0.0],["15:54:00Z",0.0],["15:55:00Z",0.0],["15:56:00Z",5.0]...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Investigate the payments service infrastructure, concurrency limits, or downstream database performance causing high p99 latency and Lambda throttling.

Hypotheses

  • likely: slow_dependency (payments is experiencing extreme latency and throttling, causing timeouts/failures in fulfillment)
  • ruled_out: bad_deploy (no recent deployments for payments or fulfillment)
  • ruled_out: throttling due to rate limits (rate limit flag is 0)

What the investigation cost

  • 7 model calls, 27,653 input and 878 output tokens (28,531 total)
  • 15 steps, 13 s of wall clock
  • 296,333 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments