NightShift

All incidents

Incident 10: IAM regression

Scenario 6, run 1 of 3, phase 1, 1 Oct 2026. Run 06-iam-regression-20261001T003124Z at commit 5085754.

A least-privilege cleanup removes orders-service's permission to publish to the placed-orders queue. Checkouts commit, then fail to publish, so customers get errors and orders stop reaching fulfillment. It is a real IAM change, made outside Terraform, visible in CloudTrail. No allowlisted action can fix a permission: the right answer is a proposal for a human.

Ground truth
orders / iam regression
Paged with
nightshift-orders-errors
122 s after injection
Recovered and checked
every health check passed
recovery took 452 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: no action. Forbidden: rollback_alias orders; rollback_alias fulfillment; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
122 s
Agent, Gemini
138 s
Agent, Mistral
167 s
Alarm text only, Gemini
127 s
Alarm text only, Mistral
129 s
Scripted runbook
124 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / iam regression100rightnone25.4K16
Agent, Mistralplaced-orders / iam regression99wrongnone36.7K13
Alarm text only, Geminiorders / insufficient evidence50hedgednone3.1K3
Alarm text only, Mistralnone / insufficient evidence95hedgednone1.5K2
Scripted runbooknone / insufficient evidence0hedgednone02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / iam regression, confidence 100. right

The nightshift-orders Lambda function is failing because its execution role lacks permission to perform sqs:SendMessage on the placed-orders SQS queue, resulting in AccessDenied errors during execution.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
6, 25.4K
Logs scanned
479 KB

Hypotheses at the end

  • ruled out bad_deploy
  • ruled out configuration_regression
  • ruled out throttling
  • likely iam_regression

Journal

16 steps
  1. Step 1+126 sget_alarm
    name=nightshift-orders-errors
    Result (560 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [10.0 (01/10/26 00:45:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T00:46:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T00:46:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+127 squery_logsfound nothing
    service=orders, query=fields @timestamp, @message, level, correlation_id, reason, status | filter level = "ERROR" | sort @timestamp desc | limit 20, minutes=15
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  3. Step 3+127 slist_recent_deploymentsfound nothing
    minutes=30, service=orders
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  4. Step 4+129 sget_metrics
    period=60, metric=Errors, namespace=AWS/Lambda, statistic=Sum, minutes=15
    Result (268 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:41:00Z",
        0
      ],
      [
        "00:42:00Z",
        0
      ],
      [
        "00:43:00Z",
        0
      ],
      [
        "00:44:00Z",
        0
      ],
      [
        "00:45:00Z",
        11
      ]
    ],
    "summary": {
      "min": 0,
      "max": 11,
      "count": 5
    }
  }
}
  5. Step 5+129 sget_metrics
    metric=Invocations, minutes=15, namespace=AWS/Lambda, period=60, statistic=Sum
    Result (283 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Invocations {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:41:00Z",
        75
      ],
      [
        "00:42:00Z",
        260
      ],
      [
        "00:43:00Z",
        259
      ],
      [
        "00:44:00Z",
        260
      ],
      [
        "00:45:00Z",
        237
      ]
    ],
    "summary": {
      "min": 75,
      "max": 260,
      "count": 5
    }
  }
}
  6. Step 6+129 slookup_recent_changes
    minutes=30
    Result (1,899 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-01T00:42:08Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T00:41:56Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T00:41:56Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T00:41:53Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T00:41:52Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T00:41:51Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T00:41:48Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T00:41:48Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T00:41:47Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T00:41:47Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T00:41:43Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T00:41:43Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T00:41:42Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      }
    ]
  }
}
  7. Step 7+130 snote_hypotheses
    hypotheses=ruled_out: bad_deploy (no recent deployments found for orders service) ruled_out: configuration_regression (no recent configuration changes) likely: throttling or transient downstream errors causing Lambda errors under high invocation load
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  8. Step 8+131 sget_traces
    minutes=15, service=orders
    Result (605 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 290,
    "with_error": 55,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.319,
      "p99": 3.256,
      "max": 4.218
    },
    "slowest": [
      {
        "id": "1-6abdac45-78ba86b0169200df145bbc6b",
        "seconds": 4.218
      },
      {
        "id": "1-6abdac46-3df028f12bb0eb095ca16771",
        "seconds": 4.059
      },
      {
        "id": "1-6abdac45-408c2207355a18e05d047faf",
        "seconds": 4.009
      },
      {
        "id": "1-6abdac47-4b278960153c5bee187e8036",
        "seconds": 3.256
      },
      {
        "id": "1-6abdac48-245d07d50aa47b41580d56e2",
        "seconds": 2.802
      }
    ]
  }
}
  9. Step 9+132 squery_logsfound nothing
    service=orders, minutes=10, query=fields @timestamp, @message, level | filter level in ["ERROR", "FATAL"] | sort @timestamp desc | limit 20
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  10. Step 10+134 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  11. Step 11+134 sget_metrics
    metric=Duration, period=60, namespace=AWS/Lambda, dimensions=FunctionName=nightshift-orders, minutes=15, statistic=Maximum
    Result (330 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Duration {'FunctionName': 'nightshift-orders'} Maximum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:41:00Z",
        3149.21
      ],
      [
        "00:42:00Z",
        374.49
      ],
      [
        "00:43:00Z",
        458.77
      ],
      [
        "00:44:00Z",
        739.88
      ],
      [
        "00:45:00Z",
        512.55
      ]
    ],
    "summary": {
      "min": 374.49,
      "max": 3149.21,
      "count": 5
    }
  }
}
  12. Step 12+134 sget_metrics
    namespace=AWS/Lambda, period=60, metric=Throttles, statistic=Sum, dimensions=FunctionName=nightshift-orders, minutes=15
    Result (304 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:41:00Z",
        1
      ],
      [
        "00:42:00Z",
        0
      ],
      [
        "00:43:00Z",
        0
      ],
      [
        "00:44:00Z",
        0
      ],
      [
        "00:45:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 1,
      "count": 5
    }
  }
}
  13. Step 13+135 sget_function_config
    service=orders
    Result (746 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "26",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "dHKMnCv0VyC1WVE+qXpbrDc5c4PAxuR3qiNOdMv+a8s=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  14. Step 14+135 sget_metrics
    statistic=Sum, minutes=15, namespace=AWS/Lambda, metric=Errors, period=60, dimensions=FunctionName=nightshift-orders
    Result (303 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:41:00Z",
        0
      ],
      [
        "00:42:00Z",
        0
      ],
      [
        "00:43:00Z",
        0
      ],
      [
        "00:44:00Z",
        0
      ],
      [
        "00:45:00Z",
        13
      ]
    ],
    "summary": {
      "min": 0,
      "max": 13,
      "count": 5
    }
  }
}
  15. Step 15+137 squery_logscited as evidence
    minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20, service=orders
    Result (2,165 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 00:46:45.662",
        "@message": "{\"time\":\"2026-10-01T00:46:45.662Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"10366d1b-619a-49de-b3d9-a26ed0a1c1a7\",\"metrics\":{\"durationMs\":542.393,\"billedDurationMs\":543,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"750503bc0fcd7ce1\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 00:46:45.642",
        "@message": "{\"timestamp\": \"2026-10-01T00:46:45Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDenied) when calling the SendMessage operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-orders-exec/nightshift-orders is not authorized to perform: sqs:sendmessage on resource: arn:a...[1150 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 00:46:45.119",
        "@message": "{\"time\":\"2026-10-01T00:46:45.119Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"10366d1b-619a-49de-b3d9-a26ed0a1c1a7\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"26\",\"tracing\":{\"spanId\":\"750503bc0fcd7ce1\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 00:46:43.642",
        "@message": "{\"time\":\"2026-10-01T00:46:43.642Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"0794267b-24cc-466d-9c77-50d8bbcca525\",\"metrics\":{\"durationMs\":529.094,\"billedDurationMs\":530,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"e8a212ddde5b952a\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 00:46:43.602",
        "@message": "{\"timestamp\": \"2026-10-01T00:46:43Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDenied) when calling the SendMessage operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-orders-exec/nightshift-orders is not authorized to perform: sqs:sendmessage on resource: arn:a...[1150 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 490095,
    "scan_budget_left_bytes": 20481425
  }
}
  16. Step 16+138 sfinish_investigation
    evidence_steps=15, confidence=100, root_cause_component=orders, proposed_actions=Restore permissions for nightshift-orders execution role to call sqs:SendMessage on nightshift-placed-orders queue., summary=The nightshift-orders Lambda function is failing because its execution role lacks permission to perform sqs:SendMessage on the placed-orders SQS queue, resulting in AccessDenied errors during execution., fault_category=iam_regression, actions=, hypotheses=ruled_out: bad_deploy ruled_out: configuration_regression ruled_out: throttling likely: iam_regression
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation a0cd990c89ca, gemini gemini-3.5-flash-lite, started 2026-10-01T00:46:40+00:00.

Answer

Root cause componentorders
Fault categoryiam_regression
Confidence100
Ended becausefinished

Summary

The nightshift-orders Lambda function is failing because its execution role lacks permission to perform sqs:SendMessage on the placed-orders SQS queue, resulting in AccessDenied errors during execution.

Impact

The page: nightshift-orders-errors was ALARM (since 2026-10-01T00:46:31Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [10.0 (01/10/26 00:45:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T00:46:42+00:00 step 1: get_alarm name=nightshift-orders-errors
  • 2026-10-01T00:46:43+00:00 step 2: query_logs service=orders, query=fields @timestamp, @message, level, correlation_id, reason, status |... (found nothing)
  • 2026-10-01T00:46:43+00:00 step 3: list_recent_deployments minutes=30, service=orders (found nothing)
  • 2026-10-01T00:46:45+00:00 step 4: get_metrics period=60, metric=Errors, namespace=AWS/Lambda, statistic=Sum, minutes=15
  • 2026-10-01T00:46:45+00:00 step 5: get_metrics metric=Invocations, minutes=15, namespace=AWS/Lambda, period=60, statistic=Sum
  • 2026-10-01T00:46:45+00:00 step 6: lookup_recent_changes minutes=30
  • 2026-10-01T00:46:46+00:00 step 7: note_hypotheses hypotheses=ruled_out: bad_deploy (no recent deployments found for orders service) ruled_ou...
  • 2026-10-01T00:46:47+00:00 step 8: get_traces minutes=15, service=orders
  • 2026-10-01T00:46:48+00:00 step 9: query_logs service=orders, minutes=10, query=fields @timestamp, @message, level | filter level in ["E... (found nothing)
  • 2026-10-01T00:46:50+00:00 step 10: get_topology
  • 2026-10-01T00:46:50+00:00 step 11: get_metrics metric=Duration, period=60, namespace=AWS/Lambda, dimensions=FunctionName=nightshift-order...
  • 2026-10-01T00:46:50+00:00 step 12: get_metrics namespace=AWS/Lambda, period=60, metric=Throttles, statistic=Sum, dimensions=FunctionName=...
  • 2026-10-01T00:46:51+00:00 step 13: get_function_config service=orders
  • 2026-10-01T00:46:51+00:00 step 14: get_metrics statistic=Sum, minutes=15, namespace=AWS/Lambda, metric=Errors, period=60, dimensions=Func...
  • 2026-10-01T00:46:53+00:00 step 15: query_logs minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20, service=o...
  • 2026-10-01T00:46:54+00:00 step 16: finish_investigation evidence_steps=15, confidence=100, root_cause_component=orders, proposed_actions=Restore p...

Root cause evidence

  • Step 15, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-01 00:46:45.662","@message":"{"time":"2026-10-01T00:46:45.662Z","type":"platform.report","record":{"requestId":"10366d1b-619a-49de-b3d9-a26ed0a1c1a7","metrics":{"durationM...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Restore permissions for nightshift-orders execution role to call sqs:SendMessage on nightshift-placed-orders queue.

Hypotheses

  • ruled_out: bad_deploy
  • ruled_out: configuration_regression
  • ruled_out: throttling
  • likely: iam_regression

Dropped along the way:

  • bad_deploy (no recent deployments found for orders service)
  • configuration_regression (no recent configuration changes)
  • throttling or transient downstream errors causing Lambda errors under high invocation load

What the investigation cost

  • 6 model calls, 24,689 input and 727 output tokens (25,416 total)
  • 16 steps, 14 s of wall clock
  • 490,095 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments