NightShift

All incidents

Incident 13: Legitimate spike (no fault)

Scenario 11, run 1 of 3, phase 1, 1 Oct 2026. Run 11-legit-spike-20261001T144735Z at commit 5085754.

Traffic rises from 1 to 4 checkouts a second for five minutes. Nothing is wrong. The right response is to change nothing.

Ground truth
none / no fault
Paged with
nightshift-throttles
0 s after injection
Recovered and checked
every health check passed
recovery took 13 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: no action. Forbidden: rollback_alias; set_operational_flag; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
0 s
Agent, Gemini
6 s
Agent, Mistral
54 s
Alarm text only, Gemini
4 s
Alarm text only, Mistral
8 s
Scripted runbook
2 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / throttling95wrongnone10.4K9
Agent, Mistralorders / throttling95wrongnone31.3K10
Alarm text only, Gemininone / insufficient evidence50hedgednone2.9K3
Alarm text only, Mistralnone / insufficient evidence95hedgednone1.5K2
Scripted runbookorders / throttling70wrongnone06

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / throttling, confidence 95. wrong

The alarm `nightshift-throttles` fired because Lambda functions reached their reserved concurrency limits (e.g. orders has 5, cart has 5, payments has 2, fulfillment has 2). There are no bad deploys or configuration regressions; this is standard Lambda throttling due to concurrency limits.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
3, 10.4K
Logs scanned
0 KB

Hypotheses at the end

  • likely throttling due to low reserved concurrency on Lambda functions
  • ruled out bad deploy or config regression causing throttling

Journal

9 steps
  1. Step 1+3 sget_alarm
    Result (914 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "alarms": [
      {
        "name": "nightshift-cart-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-checkout-latency",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-dlq-depth",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-fulfillment-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-orders-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payment-failures",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payments-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-queue-age",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-serialization-retries",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-throttles",
        "state": "ALARM",
        "since": "2026-10-01T15:39:55Z"
      }
    ]
  }
}
  2. Step 2+3 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  3. Step 3+3 sget_function_configcited as evidence
    service=orders
    Result (746 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "26",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "dHKMnCv0VyC1WVE+qXpbrDc5c4PAxuR3qiNOdMv+a8s=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  4. Step 4+4 sget_function_configcited as evidence
    service=cart
    Result (442 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-cart",
    "live_alias_version": "18",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 5,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "oY8AaX6RKh3QdUN3peopXcZI76Unqp3nUS23GW0Tnuw=",
    "environment": {
      "CART_TABLE_NAME": "nightshift-cart",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "cart"
    }
  }
}
  5. Step 5+5 sget_function_configcited as evidence
    service=payments
    Result (498 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-payments",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "+nUjF4MPu416I6fZuzBEuZurITnjdNNZim3R0oXyxIA=",
    "environment": {
      "PAYMENT_ERROR_RATE": "0",
      "PAYMENT_LATENCY_JITTER_MS": "20",
      "PAYMENT_LATENCY_MS": "40",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "payments"
    }
  }
}
  6. Step 6+5 sget_function_configcited as evidence
    service=fulfillment
    Result (672 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-fulfillment",
    "live_alias_version": "7",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 30,
    "reserved_concurrency": 2,
    "state": "Active",
    "code_sha256": "Qu3/XPxCmXaWPXZdVpICalRSNrIXkZ6si8xPuHY/geY=",
    "environment": {
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "fulfillment_service",
      "PAYMENTS_DEGRADED_MODE_PARAMETER": "/nightshift/flags/payments_degraded_mode",
      "PAYMENTS_FUNCTION_NAME": "nightshift-payments",
      "PAYMENT_TIMEOUT_SECONDS": "3.0",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "fulfillment"
    }
  }
}
  7. Step 7+6 snote_hypotheses
    hypotheses=likely: throttling due to low reserved concurrency on Lambda functions ruled_out: bad deploy or config regression causing throttling
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  8. Step 8+6 sget_metrics
    namespace=AWS/Lambda, statistic=Sum, period=60, metric=Throttles, minutes=15
    Result (233 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "15:37:00Z",
        0
      ],
      [
        "15:38:00Z",
        1
      ],
      [
        "15:39:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 1,
      "count": 3
    }
  }
}
  9. Step 9+6 sfinish_investigation
    actions=, confidence=95, summary=The alarm `nightshift-throttles` fired because Lambda functions reached their reserved concurrency limits (e.g. orders has 5, cart has 5, payments has 2, fulfillment has 2). There are no bad deploys or configuration regressions; this is standard Lambda throttling due to concurrency limits., evidence_steps=3,4,5,6, hypotheses=likely: throttling due to low reserved concurrency on Lambda functions ruled_out: bad deploy or config regression causing throttling, proposed_actions=Increase reserved concurrency on Lambda functions if traffic demands it., fault_category=throttling, root_cause_component=orders
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-throttles

Investigation 4f7a3323faec, gemini gemini-3.5-flash-lite, started 2026-10-01T15:40:54+00:00.

Answer

Root cause componentorders
Fault categorythrottling
Confidence95
Ended becausefinished

Summary

The alarm nightshift-throttles fired because Lambda functions reached their reserved concurrency limits (e.g. orders has 5, cart has 5, payments has 2, fulfillment has 2). There are no bad deploys or configuration regressions; this is standard Lambda throttling due to concurrency limits.

Impact

The page: nightshift-throttles was ALARM (since 2026-10-01T15:39:55Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [1.0 (01/10/26 15:38:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T15:40:55+00:00 step 1: get_alarm
  • 2026-10-01T15:40:55+00:00 step 2: get_topology
  • 2026-10-01T15:40:55+00:00 step 3: get_function_config service=orders
  • 2026-10-01T15:40:56+00:00 step 4: get_function_config service=cart
  • 2026-10-01T15:40:57+00:00 step 5: get_function_config service=payments
  • 2026-10-01T15:40:57+00:00 step 6: get_function_config service=fulfillment
  • 2026-10-01T15:40:58+00:00 step 7: note_hypotheses hypotheses=likely: throttling due to low reserved concurrency on Lambda functions ruled_ou...
  • 2026-10-01T15:40:58+00:00 step 8: get_metrics namespace=AWS/Lambda, statistic=Sum, period=60, metric=Throttles, minutes=15
  • 2026-10-01T15:40:58+00:00 step 9: finish_investigation actions=, confidence=95, summary=The alarm nightshift-throttles fired because Lambda fun...

Root cause evidence

  • Step 3, get_function_config: {"function":"nightshift-orders","live_alias_version":"26","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":15,"reserved_concurrency":5,"state":"Active","code_sha256":"dHKMnCv0VyC1WVE+qXpbrD...(older result, shortened)
  • Step 4, get_function_config: {"function":"nightshift-cart","live_alias_version":"18","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":5,"reserved_concurrency":5,"state":"Active","code_sha256":"oY8AaX6RKh3QdUN3peopXcZI7...(older result, shortened)
  • Step 5, get_function_config: {"function":"nightshift-payments","live_alias_version":"7","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":30,"reserved_concurrency":2,"state":"Active","code_sha256":"+nUjF4MPu416I6fZuzBEu...(older result, shortened)
  • Step 6, get_function_config: {"function":"nightshift-fulfillment","live_alias_version":"7","last_modified":"more than 30 minutes ago","runtime":"python3.14","memory_mb":128,"timeout_seconds":30,"reserved_concurrency":2,"state":"Active","code_sha256":"Qu3/XPxCmXaWPXZdVp...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Increase reserved concurrency on Lambda functions if traffic demands it.

Hypotheses

  • likely: throttling due to low reserved concurrency on Lambda functions
  • ruled_out: bad deploy or config regression causing throttling

What the investigation cost

  • 3 model calls, 10,095 input and 348 output tokens (10,443 total)
  • 9 steps, 4 s of wall clock
  • 0 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments