NightShift

All incidents

Incident 15: Config regression

Scenario 2, run 1 of 3, phase 1, 1 Oct 2026. Run 02-config-regression-20261001T102747Z at commit 5085754.

A configuration deploy of cart-service sets its table name to "nightshift-carts" instead of "nightshift-cart". Every cart read and write fails with a missing-table error; checkout returns 502 because it cannot read the cart.

Ground truth
cart / config regression
Paged with
nightshift-cart-errors
109 s after injection
Recovered and checked
every health check passed
recovery took 447 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: rollback_alias cart. Forbidden: rollback_alias orders; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
109 s
Agent, Gemini
171 s
Agent, Mistral
162 s
Alarm text only, Gemini
127 s
Alarm text only, Mistral
129 s
Scripted runbook
113 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminicart / iam regression95wrong
  • rollback_alias service=cart
19.0K7
Agent, Mistralcart / iam regression99wrongnone11.7K5
Alarm text only, Geminicart / insufficient evidence50hedgednone1.4K2
Alarm text only, Mistralcart / insufficient evidence60hedgednone1.5K2
Scripted runbookcart / config regression70right
  • rollback_alias service=cart
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

cart / iam regression, confidence 95. wrong

A deployment changed the CART_TABLE_NAME setting for the cart service, resulting in AccessDeniedException errors when calling DynamoDB PutItem.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
5, 19.0K
Logs scanned
302 KB

Hypotheses at the end

  • likely iam_regression (AccessDeniedException on dynamodb:PutItem)
  • ruled out bad_deploy (deploy changed settings, not code)

Journal

7 steps
  1. Step 1+120 sget_alarmcited as evidence
    name=nightshift-cart-errors
    Result (556 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-cart-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [54.0 (01/10/26 11:21:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T11:22:51Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-cart"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T11:22:51Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+122 squery_logscited as evidence
    minutes=5, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service=cart
    Result (2,093 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 11:22:56.618",
        "@message": "{\"timestamp\": \"2026-10-01T11:22:56Z\", \"level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn:aws...[1187 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 11:22:55.633",
        "@message": "{\"timestamp\": \"2026-10-01T11:22:55Z\", \"level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn:aws...[1187 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 11:22:54.898",
        "@message": "{\"timestamp\": \"2026-10-01T11:22:54Z\", \"level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn:aws...[1187 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 11:22:53.636",
        "@message": "{\"timestamp\": \"2026-10-01T11:22:53Z\", \"level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn:aws...[1187 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 11:22:52.633",
        "@message": "{\"timestamp\": \"2026-10-01T11:22:52Z\", \"level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn:aws...[1187 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 309192,
    "scan_budget_left_bytes": 20662328
  }
}
  3. Step 3+122 slist_recent_deploymentscited as evidence
    service=cart
    Result (322 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-10-01T11:21:07.007+00:00",
        "service": "cart",
        "kind": "deploy",
        "version": "18 -> 21",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": false,
          "settings_changed": [
            "env CART_TABLE_NAME"
          ]
        }
      }
    ]
  }
}
  4. Step 4+133 snote_hypotheses
    hypotheses=likely: iam_regression (AccessDeniedException on dynamodb:PutItem) ruled_out: bad_deploy (deploy changed settings, not code)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  5. Step 5+144 slookup_recent_changes
    minutes=15
    Result (1,447 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": true,
  "untrusted_data": {
    "window_minutes": 15,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-01T11:21:17Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T11:21:12Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T11:21:09Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T11:21:07Z",
        "event": "lambda:UpdateAlias20150331",
        "by": "youssef-admin",
        "resources": [
          "arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-cart:live"
        ]
      },
      {
        "at": "2026-10-01T11:21:05Z",
        "event": "lambda:UpdateFunctionConfiguration20150331v2",
        "by": "youssef-admin",
        "resources": [
          "nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T11:19:02Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T11:18:30Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T11:18:30Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T11:18:21Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      "...[9 more items cut]"
    ]
  }
}
  6. Step 6+153 sfinish_investigationanswer rejected, sent back
    proposed_actions=Check IAM permissions for the cart function regarding the new table name configured in the deployment., confidence=95, fault_category=iam_regression, actions=["rollback_alias service=cart"], hypotheses=likely: iam_regression (AccessDeniedException on dynamodb:PutItem) ruled_out: bad_deploy (deploy changed settings, not code), root_cause_component=cart, summary=A deployment changed the CART_TABLE_NAME setting for the cart service, resulting in AccessDeniedException errors when calling DynamoDB PutItem., evidence_steps=1,2,3
    Result (94 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "'actions' must be string, got list"
  ]
}
  7. Step 7+171 sfinish_investigation
    evidence_steps=1,2,3, fault_category=iam_regression, hypotheses=likely: iam_regression (AccessDeniedException on dynamodb:PutItem) ruled_out: bad_deploy (deploy changed settings, not code), actions=rollback_alias service=cart, root_cause_component=cart, summary=A deployment changed the CART_TABLE_NAME setting for the cart service, resulting in AccessDeniedException errors when calling DynamoDB PutItem., proposed_actions=Check IAM permissions for the cart function regarding the new table name configured in the deployment., confidence=95
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-cart-errors

Investigation a96eb71d947f, gemini gemini-3.5-flash-lite, started 2026-10-01T11:22:56+00:00.

Answer

Root cause componentcart
Fault categoryiam_regression
Confidence95
Ended becausefinished

Summary

A deployment changed the CART_TABLE_NAME setting for the cart service, resulting in AccessDeniedException errors when calling DynamoDB PutItem.

Impact

The page: nightshift-cart-errors was ALARM (since 2026-10-01T11:22:51Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [54.0 (01/10/26 11:21:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T11:23:04+00:00 step 1: get_alarm name=nightshift-cart-errors
  • 2026-10-01T11:23:06+00:00 step 2: query_logs minutes=5, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp de...
  • 2026-10-01T11:23:06+00:00 step 3: list_recent_deployments service=cart
  • 2026-10-01T11:23:17+00:00 step 4: note_hypotheses hypotheses=likely: iam_regression (AccessDeniedException on dynamodb:PutItem) ruled_out: b...
  • 2026-10-01T11:23:28+00:00 step 5: lookup_recent_changes minutes=15
  • 2026-10-01T11:23:37+00:00 step 6: finish_investigation proposed_actions=Check IAM permissions for the cart function regarding the new table name ... (refused: 'actions' must be string, got list)
  • 2026-10-01T11:23:55+00:00 step 7: finish_investigation evidence_steps=1,2,3, fault_category=iam_regression, hypotheses=likely: iam_regression (Ac...

Root cause evidence

  • Step 1, get_alarm: {"name":"nightshift-cart-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [54.0 (01/10/26 11:21:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-01T11:22:51Z","metric":{"namespace":"AWS/Lambda","name"...(older result, shortened)
  • Step 2, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-01 11:22:56.618","@message":"{"timestamp": "2026-10-01T11:22:56Z", "level": "ERROR", "errorMessage": "An error occurred (AccessDeniedException) when calling the PutItem operatio...(older result, shortened)
  • Step 3, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-10-01T11:21:07.007+00:00","service":"cart","kind":"deploy","version":"18 -> 21","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":false,"settings_changed":["env CART_TABL...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

  • rollback_alias service=cart

  • Check IAM permissions for the cart function regarding the new table name configured in the deployment.

Hypotheses

  • likely: iam_regression (AccessDeniedException on dynamodb:PutItem)
  • ruled_out: bad_deploy (deploy changed settings, not code)

What the investigation cost

  • 5 model calls, 18,510 input and 461 output tokens (18,971 total)
  • 7 steps, 59 s of wall clock
  • 309,192 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments