NightShift

All incidents

Incident 8: Throttling

Scenario 9, run 1 of 3, phase 1, 30 Sep 2026. Run 09-throttling-20260930T215659Z at commit 5085754.

Someone lowers cart-service's reserved concurrency from 5 to 1 from the console, and traffic rises from 1 to 3 checkouts a second. cart is throttled; orders turns the throttles into 502s. The change is one PutFunctionConcurrency call: no deploy, no new version.

Ground truth
cart / throttling
Paged with
nightshift-throttles
107 s after injection
Recovered and checked
every health check passed
recovery took 471 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: set_operational_flag checkout_rate_limit. Forbidden: rollback_alias; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
107 s
Agent, Gemini
116 s
Agent, Mistral
132 s
Alarm text only, Gemini
110 s
Alarm text only, Mistral
124 s
Scripted runbook
109 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminicart / throttling95right
  • set_operational_flag name=checkout_rate_limit value=10
23.6K15
Agent, Mistralnone / no fault95wrongnone17.0K7
Alarm text only, Geminipayments / throttling90wrong
  • set_operational_flag name=payments_degraded_mode value=true
1.4K2
Alarm text only, Mistraldsql / throttling85wrongnone3.6K3
Scripted runbookcart / throttling70rightnone06

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

cart / throttling, confidence 95. right

The nightshift-throttles alarm triggered due to the nightshift-cart Lambda function hitting its reserved concurrency limit (reserved concurrency of 1), resulting in 41 throttles as overall invocations spiked.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
6, 23.6K
Logs scanned
0 KB

Hypotheses at the end

  • likely reserved concurrency limits are being reached causing throttling
  • ruled out bad deploy caused the throttles

Journal

15 steps
  1. Step 1+110 sget_alarm
    Result (914 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "alarms": [
      {
        "name": "nightshift-cart-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-checkout-latency",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-dlq-depth",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-fulfillment-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-orders-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payment-failures",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payments-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-queue-age",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-serialization-retries",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-throttles",
        "state": "ALARM",
        "since": "2026-09-30T22:46:55Z"
      }
    ]
  }
}
  2. Step 2+110 sget_function_config
    service=orders
    Result (746 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "26",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "dHKMnCv0VyC1WVE+qXpbrDc5c4PAxuR3qiNOdMv+a8s=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  3. Step 3+110 sget_function_config
    service=payments
    Result (498 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-payments",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "+nUjF4MPu416I6fZuzBEuZurITnjdNNZim3R0oXyxIA=",
    "environment": {
      "PAYMENT_ERROR_RATE": "0",
      "PAYMENT_LATENCY_JITTER_MS": "20",
      "PAYMENT_LATENCY_MS": "40",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "payments"
    }
  }
}
  4. Step 4+111 sget_function_configcited as evidence
    service=cart
    Result (442 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-cart",
    "live_alias_version": "18",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 5,
    "reserved_concurrency": 1,
    "state": "Active",
    "code_sha256": "oY8AaX6RKh3QdUN3peopXcZI76Unqp3nUS23GW0Tnuw=",
    "environment": {
      "CART_TABLE_NAME": "nightshift-cart",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "cart"
    }
  }
}
  5. Step 5+112 sget_function_config
    service=fulfillment
    Result (672 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-fulfillment",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "Qu3/XPxCmXaWPXZdVpICalRSNrIXkZ6si8xPuHY/geY=",
    "environment": {
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "fulfillment_service",
      "PAYMENTS_DEGRADED_MODE_PARAMETER": "/nightshift/flags/payments_degraded_mode",
      "PAYMENTS_FUNCTION_NAME": "nightshift-payments",
      "PAYMENT_TIMEOUT_SECONDS": "3.0",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "fulfillment"
    }
  }
}
  6. Step 6+112 sget_function_configtool returned an error
    service=hello
    Result (153 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "error": "unknown service 'hello'; have ['cart', 'fulfillment', 'orders', 'payments']"
  }
}
  7. Step 7+113 sget_metricscited as evidence
    metric=Throttles, namespace=AWS/Lambda, statistic=Sum, minutes=15
    Result (272 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "22:42:00Z",
        0
      ],
      [
        "22:43:00Z",
        0
      ],
      [
        "22:44:00Z",
        0
      ],
      [
        "22:45:00Z",
        41
      ],
      [
        "22:46:00Z",
        37
      ]
    ],
    "summary": {
      "min": 0,
      "max": 41,
      "count": 5
    }
  }
}
  8. Step 8+113 sget_metricscited as evidence
    metric=Invocations, statistic=Sum, namespace=AWS/Lambda, minutes=15
    Result (285 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Invocations {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "22:42:00Z",
        164
      ],
      [
        "22:43:00Z",
        260
      ],
      [
        "22:44:00Z",
        257
      ],
      [
        "22:45:00Z",
        408
      ],
      [
        "22:46:00Z",
        303
      ]
    ],
    "summary": {
      "min": 164,
      "max": 408,
      "count": 5
    }
  }
}
  9. Step 9+113 snote_hypotheses
    hypotheses=likely: reserved concurrency limits are being reached causing throttling ruled_out: bad deploy caused the throttles
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  10. Step 10+114 sget_metrics
    metric=Throttles, minutes=15, namespace=AWS/Lambda, dimensions=FunctionName=nightshift-orders, statistic=Sum
    Result (304 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "22:42:00Z",
        0
      ],
      [
        "22:43:00Z",
        0
      ],
      [
        "22:44:00Z",
        0
      ],
      [
        "22:45:00Z",
        0
      ],
      [
        "22:46:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 5
    }
  }
}
  11. Step 11+114 sfinish_investigationanswer rejected, sent back
    evidence_steps=2,3,4,5,7,8, root_cause_component=orders, actions=set_operational_flag name=checkout_rate_limit value=false, confidence=95, hypotheses=likely: reserved concurrency limits are being reached causing throttling ruled_out: bad deploy caused the throttles, proposed_actions=Increase reserved concurrency or adjust checkout rate limits if traffic increases are expected., fault_category=throttling, summary=The nightshift-throttles alarm triggered due to Lambda function invocations hitting reserved concurrency limits (reaching 41 throttles at 22:45:00Z) as overall invocations spiked to 408.
    Result (180 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "action 'set_operational_flag name=checkout_rate_limit value=false': 'false' is not a valid value for checkout_rate_limit"
  ]
}
  12. Step 12+115 sget_metricscited as evidence
    statistic=Sum, dimensions=FunctionName=nightshift-cart, metric=Throttles, minutes=15, namespace=AWS/Lambda
    Result (305 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-cart'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "22:42:00Z",
        0
      ],
      [
        "22:43:00Z",
        0
      ],
      [
        "22:44:00Z",
        0
      ],
      [
        "22:45:00Z",
        41
      ],
      [
        "22:46:00Z",
        42
      ]
    ],
    "summary": {
      "min": 0,
      "max": 42,
      "count": 5
    }
  }
}
  13. Step 13+115 sget_metrics
    namespace=AWS/Lambda, dimensions=FunctionName=nightshift-payments, minutes=15, statistic=Sum, metric=Throttles
    Result (306 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-payments'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "22:42:00Z",
        0
      ],
      [
        "22:43:00Z",
        0
      ],
      [
        "22:44:00Z",
        0
      ],
      [
        "22:45:00Z",
        0
      ],
      [
        "22:46:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 5
    }
  }
}
  14. Step 14+115 sget_metrics
    namespace=AWS/Lambda, dimensions=FunctionName=nightshift-fulfillment, minutes=15, metric=Throttles, statistic=Sum
    Result (309 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-fulfillment'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "22:42:00Z",
        0
      ],
      [
        "22:43:00Z",
        0
      ],
      [
        "22:44:00Z",
        0
      ],
      [
        "22:45:00Z",
        0
      ],
      [
        "22:46:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 5
    }
  }
}
  15. Step 15+116 sfinish_investigation
    root_cause_component=cart, confidence=95, summary=The nightshift-throttles alarm triggered due to the nightshift-cart Lambda function hitting its reserved concurrency limit (reserved concurrency of 1), resulting in 41 throttles as overall invocations spiked., proposed_actions=Increase reserved concurrency or adjust rate limits., fault_category=throttling, evidence_steps=4,7,8,12, actions=set_operational_flag name=checkout_rate_limit value=10, hypotheses=likely: reserved concurrency limits are being reached causing throttling ruled_out: bad deploy caused the throttles
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-throttles

Investigation fa1a726db0a1, gemini gemini-3.5-flash-lite, started 2026-09-30T22:47:03+00:00.

Answer

Root cause componentcart
Fault categorythrottling
Confidence95
Ended becausefinished

Summary

The nightshift-throttles alarm triggered due to the nightshift-cart Lambda function hitting its reserved concurrency limit (reserved concurrency of 1), resulting in 41 throttles as overall invocations spiked.

Impact

The page: nightshift-throttles was ALARM (since 2026-09-30T22:46:55Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [41.0 (30/09/26 22:45:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-09-30T22:47:05+00:00 step 1: get_alarm
  • 2026-09-30T22:47:05+00:00 step 2: get_function_config service=orders
  • 2026-09-30T22:47:05+00:00 step 3: get_function_config service=payments
  • 2026-09-30T22:47:06+00:00 step 4: get_function_config service=cart
  • 2026-09-30T22:47:07+00:00 step 5: get_function_config service=fulfillment
  • 2026-09-30T22:47:07+00:00 step 6: get_function_config service=hello (failed)
  • 2026-09-30T22:47:08+00:00 step 7: get_metrics metric=Throttles, namespace=AWS/Lambda, statistic=Sum, minutes=15
  • 2026-09-30T22:47:08+00:00 step 8: get_metrics metric=Invocations, statistic=Sum, namespace=AWS/Lambda, minutes=15
  • 2026-09-30T22:47:08+00:00 step 9: note_hypotheses hypotheses=likely: reserved concurrency limits are being reached causing throttling ruled_...
  • 2026-09-30T22:47:09+00:00 step 10: get_metrics metric=Throttles, minutes=15, namespace=AWS/Lambda, dimensions=FunctionName=nightshift-ord...
  • 2026-09-30T22:47:09+00:00 step 11: finish_investigation evidence_steps=2,3,4,5,7,8, root_cause_component=orders, actions=set_operational_flag name... (refused: action 'set_operational_flag name=checkout_rate_limit value=false': 'false' is not a valid value for checkout_rate_limit)
  • 2026-09-30T22:47:10+00:00 step 12: get_metrics statistic=Sum, dimensions=FunctionName=nightshift-cart, metric=Throttles, minutes=15, name...
  • 2026-09-30T22:47:10+00:00 step 13: get_metrics namespace=AWS/Lambda, dimensions=FunctionName=nightshift-payments, minutes=15, statistic=S...
  • 2026-09-30T22:47:10+00:00 step 14: get_metrics namespace=AWS/Lambda, dimensions=FunctionName=nightshift-fulfillment, minutes=15, metric=T...
  • 2026-09-30T22:47:11+00:00 step 15: finish_investigation root_cause_component=cart, confidence=95, summary=The nightshift-throttles alarm triggered...

Root cause evidence

  • Step 4, get_function_config: {"function":"nightshift-cart","live_alias_version":"18","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":5,"reserved_concurrency":1,"state":"Active","code_sha256":"oY8AaX6RKh3QdUN3peopXcZI7...(older result, shortened)
  • Step 7, get_metrics: {"metric":"AWS/Lambda Throttles {} Sum per 60s","window_minutes":15,"points":[["22:42:00Z",0.0],["22:43:00Z",0.0],["22:44:00Z",0.0],["22:45:00Z",41.0],["22:46:00Z",37.0]],"summary":{"min":0.0,"max":41.0,"count":5}}
  • Step 8, get_metrics: {"metric":"AWS/Lambda Invocations {} Sum per 60s","window_minutes":15,"points":[["22:42:00Z",164.0],["22:43:00Z",260.0],["22:44:00Z",257.0],["22:45:00Z",408.0],["22:46:00Z",303.0]],"summary":{"min":164.0,"max":408.0,"count":5}}
  • Step 12, get_metrics: {"metric":"AWS/Lambda Throttles {'FunctionName': 'nightshift-cart'} Sum per 60s","window_minutes":15,"points":[["22:42:00Z",0.0],["22:43:00Z",0.0],["22:44:00Z",0.0],["22:45:00Z",41.0],["22:46:00Z",42.0]],"summary":{"min":0.0,"max":42.0,"cou...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

  • set_operational_flag name=checkout_rate_limit value=10

  • Increase reserved concurrency or adjust rate limits.

Hypotheses

  • likely: reserved concurrency limits are being reached causing throttling
  • ruled_out: bad deploy caused the throttles

What the investigation cost

  • 6 model calls, 22,858 input and 746 output tokens (23,604 total)
  • 15 steps, 8 s of wall clock
  • 0 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments