NightShift

All incidents

Incident 3: Legitimate spike (no fault)

Scenario 11, run 2 of 3, phase 1, 30 Sep 2026. Run 11-legit-spike-20260930T165050Z at commit 5085754.

Traffic rises from 1 to 4 checkouts a second for five minutes. Nothing is wrong. The right response is to change nothing.

Ground truth
none / no fault
Paged with
nightshift-serialization-retries
128 s after injection
Recovered and checked
every health check passed
recovery took 306 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: no action. Forbidden: rollback_alias; set_operational_flag; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
128 s
Agent, Gemini
137 s
Agent, Mistral
163 s
Alarm text only, Gemini
131 s
Alarm text only, Mistral
135 s
Scripted runbook
130 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / hot row contention95wrongnone18.0K11
Agent, Mistralorders / hot row contention95wrongnone26.8K10
Alarm text only, Geminiorders / hot row contention90wrongnone1.4K2
Alarm text only, Mistralorders / insufficient evidence60hedgednone1.5K2
Scripted runbookdsql / hot row contention70wrongnone02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / hot row contention, confidence 95. wrong

The `nightshift-serialization-retries` alarm fired because the `SerializationRetries` metric for the orders service spiked to 23 retries per minute. There were no recent deployments, errors, or trace faults, indicating that many concurrent transactions were attempting to update the same database rows, leading to transaction serialization failures and automatic retries.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
5, 18.0K
Logs scanned
252 KB

Hypotheses at the end

  • likely hot_row_contention in orders service / database due to high serialization retries
  • ruled out bad_deploy (no recent deployments)
  • ruled out errors in logs or trace faults (traces show no errors or faults)

Journal

11 steps
  1. Step 1+131 sget_alarm
    Result (910 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "alarms": [
      {
        "name": "nightshift-cart-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-checkout-latency",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-dlq-depth",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-fulfillment-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-orders-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payment-failures",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-payments-errors",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-queue-age",
        "state": "OK",
        "since": "more than 30 minutes ago"
      },
      {
        "name": "nightshift-serialization-retries",
        "state": "ALARM",
        "since": "2026-09-30T17:42:02Z"
      },
      {
        "name": "nightshift-throttles",
        "state": "OK",
        "since": "2026-09-30T17:39:55Z"
      }
    ]
  }
}
  2. Step 2+131 sget_metricsfound nothing
    minutes=15, metric=SerializationRetries, namespace=NightShift, statistic=Sum
    Result (219 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift SerializationRetries {} Sum per 60s",
    "window_minutes": 15,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  3. Step 3+133 squery_logsfound nothing
    query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" | sort @timestamp desc, service=orders, minutes=15
    Result (145 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 258487,
    "scan_budget_left_bytes": 20713033
  }
}
  4. Step 4+134 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  5. Step 5+134 slist_recent_deploymentsfound nothing
    service=orders
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  6. Step 6+135 sget_traces
    service=orders, minutes=15
    Result (603 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 624,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.347,
      "p99": 0.83,
      "max": 4.396
    },
    "slowest": [
      {
        "id": "1-6abd48cd-3592232c728e91124748f415",
        "seconds": 4.396
      },
      {
        "id": "1-6abd48cd-3b0460bf2c3b9f69640475aa",
        "seconds": 4.056
      },
      {
        "id": "1-6abd48ce-017417dd67dbda4d451fd69a",
        "seconds": 3.387
      },
      {
        "id": "1-6abd48cf-720593e15c685869550a922c",
        "seconds": 3.156
      },
      {
        "id": "1-6abd48d0-453b401d2eb1a5d10ac1a757",
        "seconds": 2.919
      }
    ]
  }
}
  7. Step 7+134 sget_metricsfound nothing
    minutes=15, statistic=Sum, metric=SerializationRetries, namespace=NightShift
    Result (219 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift SerializationRetries {} Sum per 60s",
    "window_minutes": 15,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  8. Step 8+134 sget_metricscited as evidence
    statistic=Sum, metric=SerializationRetries, minutes=15, namespace=NightShift, dimensions=service=orders
    Result (266 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift SerializationRetries {'service': 'orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "17:37:00Z",
        1
      ],
      [
        "17:40:00Z",
        23
      ],
      [
        "17:41:00Z",
        21
      ]
    ],
    "summary": {
      "min": 1,
      "max": 23,
      "count": 3
    }
  }
}
  9. Step 9+134 sget_metricsfound nothing
    dimensions=service=orders, minutes=15, statistic=Sum, namespace=NightShift, metric=OrdersPaid
    Result (228 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift OrdersPaid {'service': 'orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  10. Step 10+135 snote_hypotheses
    hypotheses=likely: hot_row_contention in orders service / database due to high serialization retries ruled_out: bad_deploy (no recent deployments) ruled_out: errors in logs or trace faults (traces show no errors or faults)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  11. Step 11+137 sfinish_investigation
    proposed_actions=Investigate concurrent transactions updating the same rows or inventory/counters in the orders service., root_cause_component=orders, confidence=95, summary=The `nightshift-serialization-retries` alarm fired because the `SerializationRetries` metric for the orders service spiked to 23 retries per minute. There were no recent deployments, errors, or trace faults, indicating that many concurrent transactions were attempting to update the same database rows, leading to transaction serialization failures and automatic retries., actions=, fault_category=hot_row_contention, hypotheses=likely: hot_row_contention in orders service / database due to high serialization retries ruled_out: bad_deploy (no recent deployments) ruled_out: errors in logs or trace faults (traces show no errors or faults), evidence_steps=8
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-serialization-retries

Investigation e960136037c3, gemini gemini-3.5-flash-lite, started 2026-09-30T17:42:10+00:00.

Answer

Root cause componentorders
Fault categoryhot_row_contention
Confidence95
Ended becausefinished

Summary

The nightshift-serialization-retries alarm fired because the SerializationRetries metric for the orders service spiked to 23 retries per minute. There were no recent deployments, errors, or trace faults, indicating that many concurrent transactions were attempting to update the same database rows, leading to transaction serialization failures and automatic retries.

Impact

The page: nightshift-serialization-retries was ALARM (since 2026-09-30T17:42:02Z). Reason recorded by CloudWatch: Threshold Crossed: 2 datapoints [19.0 (30/09/26 17:41:00), 23.0 (30/09/26 17:40:00)] were greater than or equal to the threshold (10.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-09-30T17:42:11+00:00 step 1: get_alarm
  • 2026-09-30T17:42:11+00:00 step 2: get_metrics minutes=15, metric=SerializationRetries, namespace=NightShift, statistic=Sum (found nothing)
  • 2026-09-30T17:42:13+00:00 step 3: query_logs query=fields @timestamp, @message | filter level = "ERROR" or level = "WARN" | sort @times... (found nothing)
  • 2026-09-30T17:42:14+00:00 step 4: get_topology
  • 2026-09-30T17:42:14+00:00 step 5: list_recent_deployments service=orders (found nothing)
  • 2026-09-30T17:42:15+00:00 step 6: get_traces service=orders, minutes=15
  • 2026-09-30T17:42:14+00:00 step 7: get_metrics minutes=15, statistic=Sum, metric=SerializationRetries, namespace=NightShift (found nothing)
  • 2026-09-30T17:42:14+00:00 step 8: get_metrics statistic=Sum, metric=SerializationRetries, minutes=15, namespace=NightShift, dimensions=s...
  • 2026-09-30T17:42:14+00:00 step 9: get_metrics dimensions=service=orders, minutes=15, statistic=Sum, namespace=NightShift, metric=OrdersP... (found nothing)
  • 2026-09-30T17:42:15+00:00 step 10: note_hypotheses hypotheses=likely: hot_row_contention in orders service / database due to high serializati...
  • 2026-09-30T17:42:17+00:00 step 11: finish_investigation proposed_actions=Investigate concurrent transactions updating the same rows or inventory/c...

Root cause evidence

  • Step 8, get_metrics: {"metric":"NightShift SerializationRetries {'service': 'orders'} Sum per 60s","window_minutes":15,"points":[["17:37:00Z",1.0],["17:40:00Z",23.0],["17:41:00Z",21.0]],"summary":{"min":1.0,"max":23.0,"count":3}}

Proposed fix (for a human to decide; the agent changed nothing)

  • Investigate concurrent transactions updating the same rows or inventory/counters in the orders service.

Hypotheses

  • likely: hot_row_contention in orders service / database due to high serialization retries
  • ruled_out: bad_deploy (no recent deployments)
  • ruled_out: errors in logs or trace faults (traces show no errors or faults)

What the investigation cost

  • 5 model calls, 17,418 input and 548 output tokens (17,966 total)
  • 11 steps, 9 s of wall clock
  • 258,487 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments