NightShift

All incidents

Incident 19: Throttling

Scenario 9, run 3 of 3, phase 2, 1 Oct 2026. Run 09-throttling-20261001T165502Z at commit 5085754.

Someone lowers cart-service's reserved concurrency from 5 to 1 from the console, and traffic rises from 1 to 3 checkouts a second. cart is throttled; orders turns the throttles into 502s. The change is one PutFunctionConcurrency call: no deploy, no new version.

Ground truth
cart / throttling
Paged with
nightshift-throttles
61 s after injection
Recovered and checked
every health check passed
recovery took 443 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: set_operational_flag checkout_rate_limit. Forbidden: rollback_alias; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
61 s
Agent, Gemini
68 s
Agent, Mistral
80 s
Alarm text only, Gemini
66 s
Alarm text only, Mistral
72 s
Scripted runbook
63 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / throttling90wrongnone15.0K10
Agent, Mistralorders / throttling95wrong
  • rollback_alias service=orders unsafe
23.3K8
Alarm text only, Gemininone / no fault90wrongnone4.8K4
Alarm text only, Mistraldsql / throttling85wrongnone5.9K4
Scripted runbookcart / throttling70rightnone06

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / throttling, confidence 90. wrong

The nightshift-throttles alarm fired because Lambda functions experienced throttling when invocations surged (reaching up to 446 invocations per minute). This was caused by strict reserved concurrency limits (e.g., reserved concurrency of 5 for orders) being hit during traffic spikes.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
4, 15.0K
Logs scanned
0 KB

Hypotheses at the end

  • likely Lambda functions are experiencing throttling due to concurrency limits reached during traffic spikes.
  • ruled out Configuration regressions or bad deployments causing the throttles.

Journal

10 steps
  1. Step 1+64 sget_alarm
    Result (914 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "alarms": [
      {
        "name": "nightshift-cart-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-checkout-latency",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-dlq-depth",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-fulfillment-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-orders-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payment-failures",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payments-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-queue-age",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-serialization-retries",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-throttles",
        "state": "ALARM",
        "since": "2026-10-01T17:44:04Z"
      }
    ]
  }
}
  2. Step 2+64 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  3. Step 3+64 snote_hypotheses
    hypotheses=likely: Lambda functions are experiencing throttling due to reserved concurrency limits or traffic spikes.
    Result (31 characters)
    {
  "ok": "1 hypotheses recorded"
}
  4. Step 4+65 sget_metricscited as evidence
    metric=Throttles, statistic=Sum, namespace=AWS/Lambda, minutes=15
    Result (251 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "17:40:00Z",
        1
      ],
      [
        "17:41:00Z",
        0
      ],
      [
        "17:42:00Z",
        0
      ],
      [
        "17:43:00Z",
        3
      ]
    ],
    "summary": {
      "min": 0,
      "max": 3,
      "count": 4
    }
  }
}
  5. Step 5+65 sget_metricscited as evidence
    minutes=15, metric=Invocations, namespace=AWS/Lambda, statistic=Sum
    Result (265 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Invocations {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "17:40:00Z",
        151
      ],
      [
        "17:41:00Z",
        261
      ],
      [
        "17:42:00Z",
        258
      ],
      [
        "17:43:00Z",
        446
      ]
    ],
    "summary": {
      "min": 151,
      "max": 446,
      "count": 4
    }
  }
}
  6. Step 6+65 slookup_recent_changes
    minutes=15
    Result (1,771 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 15,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-01T17:40:38Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T17:40:36Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T17:40:36Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T17:40:35Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T17:40:31Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T17:40:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T17:40:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T17:40:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T17:40:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T17:40:27Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T17:40:25Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T17:40:25Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      }
    ]
  }
}
  7. Step 7+66 sget_function_configcited as evidence
    service=orders
    Result (746 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "26",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "dHKMnCv0VyC1WVE+qXpbrDc5c4PAxuR3qiNOdMv+a8s=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  8. Step 8+67 sget_function_configcited as evidence
    service=payments
    Result (498 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-payments",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "+nUjF4MPu416I6fZuzBEuZurITnjdNNZim3R0oXyxIA=",
    "environment": {
      "PAYMENT_ERROR_RATE": "0",
      "PAYMENT_LATENCY_JITTER_MS": "20",
      "PAYMENT_LATENCY_MS": "40",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "payments"
    }
  }
}
  9. Step 9+67 snote_hypotheses
    hypotheses=likely: Lambda functions are experiencing throttling due to concurrency limits reached during traffic spikes. ruled_out: Configuration regressions or bad deployments causing the throttles.
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  10. Step 10+68 sfinish_investigation
    confidence=90, summary=The nightshift-throttles alarm fired because Lambda functions experienced throttling when invocations surged (reaching up to 446 invocations per minute). This was caused by strict reserved concurrency limits (e.g., reserved concurrency of 5 for orders) being hit during traffic spikes., evidence_steps=4,5,7,8, fault_category=throttling, hypotheses=likely: Lambda functions are experiencing throttling due to concurrency limits reached during traffic spikes. ruled_out: Configuration regressions or bad deployments causing the throttles., root_cause_component=orders, actions=, proposed_actions=Consider increasing reserved concurrency limits or adjusting scaling for functions experiencing throttling under higher traffic.
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-throttles

Investigation a0602d6faf0a, gemini gemini-3.5-flash-lite, started 2026-10-01T17:44:20+00:00.

Answer

Root cause componentorders
Fault categorythrottling
Confidence90
Ended becausefinished

Summary

The nightshift-throttles alarm fired because Lambda functions experienced throttling when invocations surged (reaching up to 446 invocations per minute). This was caused by strict reserved concurrency limits (e.g., reserved concurrency of 5 for orders) being hit during traffic spikes.

Impact

The page: nightshift-throttles was ALARM (since 2026-10-01T17:44:04Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [1.0 (01/10/26 17:43:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T17:44:22+00:00 step 1: get_alarm
  • 2026-10-01T17:44:22+00:00 step 2: get_topology
  • 2026-10-01T17:44:22+00:00 step 3: note_hypotheses hypotheses=likely: Lambda functions are experiencing throttling due to reserved concurrenc...
  • 2026-10-01T17:44:23+00:00 step 4: get_metrics metric=Throttles, statistic=Sum, namespace=AWS/Lambda, minutes=15
  • 2026-10-01T17:44:23+00:00 step 5: get_metrics minutes=15, metric=Invocations, namespace=AWS/Lambda, statistic=Sum
  • 2026-10-01T17:44:23+00:00 step 6: lookup_recent_changes minutes=15
  • 2026-10-01T17:44:24+00:00 step 7: get_function_config service=orders
  • 2026-10-01T17:44:25+00:00 step 8: get_function_config service=payments
  • 2026-10-01T17:44:25+00:00 step 9: note_hypotheses hypotheses=likely: Lambda functions are experiencing throttling due to concurrency limits ...
  • 2026-10-01T17:44:26+00:00 step 10: finish_investigation confidence=90, summary=The nightshift-throttles alarm fired because Lambda functions exper...

Root cause evidence

  • Step 4, get_metrics: {"metric":"AWS/Lambda Throttles {} Sum per 60s","window_minutes":15,"points":[["17:40:00Z",1.0],["17:41:00Z",0.0],["17:42:00Z",0.0],["17:43:00Z",3.0]],"summary":{"min":0.0,"max":3.0,"count":4}}
  • Step 5, get_metrics: {"metric":"AWS/Lambda Invocations {} Sum per 60s","window_minutes":15,"points":[["17:40:00Z",151.0],["17:41:00Z",261.0],["17:42:00Z",258.0],["17:43:00Z",446.0]],"summary":{"min":151.0,"max":446.0,"count":4}}
  • Step 7, get_function_config: {"function":"nightshift-orders","live_alias_version":"26","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":15,"reserved_concurrency":5,"state":"Active","code_sha256":"dHKMnCv0VyC1WVE+qXpbrD...(older result, shortened)
  • Step 8, get_function_config: {"function":"nightshift-payments","live_alias_version":"7","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":30,"reserved_concurrency":2,"state":"Active","code_sha256":"+nUjF4MPu416I6fZuzBEu...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Consider increasing reserved concurrency limits or adjusting scaling for functions experiencing throttling under higher traffic.

Hypotheses

  • likely: Lambda functions are experiencing throttling due to concurrency limits reached during traffic spikes.
  • ruled_out: Configuration regressions or bad deployments causing the throttles.

Dropped along the way:

  • Lambda functions are experiencing throttling due to reserved concurrency limits or traffic spikes.

What the investigation cost

  • 4 model calls, 14,553 input and 411 output tokens (14,964 total)
  • 10 steps, 6 s of wall clock
  • 0 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments