NightShift

All incidents

Incident 27: IAM regression

Scenario 6, run 3 of 3, phase 2, 2 Oct 2026. Run 06-iam-regression-20261002T011656Z at commit 5085754.

A least-privilege cleanup removes orders-service's permission to publish to the placed-orders queue. Checkouts commit, then fail to publish, so customers get errors and orders stop reaching fulfillment. It is a real IAM change, made outside Terraform, visible in CloudTrail. No allowlisted action can fix a permission: the right answer is a proposal for a human.

Ground truth
orders / iam regression
Paged with
nightshift-orders-errors
169 s after injection
Recovered and checked
every health check passed
recovery took 453 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: no action. Forbidden: rollback_alias orders; rollback_alias fulfillment; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
169 s
Agent, Gemini
183 s
Agent, Mistral
201 s
Alarm text only, Gemini
174 s
Alarm text only, Mistral
177 s
Scripted runbook
171 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / iam regression100rightnone20.4K11
Agent, Mistraldsql / slow dependency85wrongnone37.1K14
Alarm text only, Geminiorders / insufficient evidence50hedgednone3.0K3
Alarm text only, Mistralorders / insufficient evidence60hedgednone1.6K2
Scripted runbooknone / insufficient evidence0hedgednone02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / iam regression, confidence 100. right

The nightshift-orders Lambda function is failing with AccessDenied errors when trying to call the SQS SendMessage operation because the execution role lacks the required sqs:sendmessage permission.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
5, 20.4K
Logs scanned
621 KB

Hypotheses at the end

  • ruled out bad_deploy
  • ruled out config_regression
  • ruled out slow_dependency
  • ruled out timeout_regression
  • ruled out throttling
  • likely iam_regression

Journal

11 steps
  1. Step 1+172 sget_alarm
    name=nightshift-orders-errors
    Result (560 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [35.0 (02/10/26 02:06:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-02T02:07:55Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-02T02:07:55Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+172 slist_recent_deploymentsfound nothing
    service=orders
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  3. Step 3+174 squery_logsfound nothing
    service=orders, query=fields @timestamp, level, message, correlation_id | filter level = "ERROR" | sort @timestamp desc, minutes=10
    Result (144 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 40302,
    "scan_budget_left_bytes": 20931218
  }
}
  4. Step 4+176 sget_traces
    minutes=10, service=orders
    Result (604 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 329,
    "with_error": 98,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.362,
      "p99": 2.111,
      "max": 4.593
    },
    "slowest": [
      {
        "id": "1-6abf10ad-55fc31cd1558753909b4c11d",
        "seconds": 4.593
      },
      {
        "id": "1-6abf10ad-5f6b455832deec363b5fb2c2",
        "seconds": 4.273
      },
      {
        "id": "1-6abf10ae-23ad58f566dd08ba52815074",
        "seconds": 3.313
      },
      {
        "id": "1-6abf10af-66865220576d45ff43876ead",
        "seconds": 2.43
      },
      {
        "id": "1-6abf10b0-588910fb3ce8f28f16bd1ecc",
        "seconds": 2.111
      }
    ]
  }
}
  5. Step 5+176 sget_function_config
    service=orders
    Result (746 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "26",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "dHKMnCv0VyC1WVE+qXpbrDc5c4PAxuR3qiNOdMv+a8s=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  6. Step 6+177 slookup_recent_changes
    minutes=15
    Result (1,899 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 15,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-02T02:02:41Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-02T02:02:39Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-02T02:02:37Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-02T02:02:34Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-02T02:02:33Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-02T02:02:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-02T02:02:27Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-02T02:02:24Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-02T02:02:24Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-02T02:02:24Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-02T02:02:24Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-02T02:02:23Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-02T02:02:22Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      }
    ]
  }
}
  7. Step 7+178 snote_hypotheses
    hypotheses=ruled_out: bad_deploy (no recent deployments) ruled_out: config_regression (no recent config changes found in CloudTrail except log stream creation) likely: slow_dependency or dsql / timeout_regression or throttling (errors and high duration seen in X-Ray traces)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  8. Step 8+180 squery_logscited as evidence
    minutes=10, service=orders, query=fields @timestamp, @message | sort @timestamp desc | limit 20
    Result (2,187 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 02:08:10.087",
        "@message": "{\"time\":\"2026-10-02T02:08:10.087Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"05e2c27a-6402-4293-a6f3-c6b2e08cf88e\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"26\",\"tracing\":{\"spanId\":\"74b5bd0913353e47\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 02:08:09.441",
        "@message": "{\"time\":\"2026-10-02T02:08:09.441Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"ae4ab5c0-6521-41ca-98d5-6fcfbc87ba41\",\"metrics\":{\"durationMs\":388.458,\"billedDurationMs\":389,\"memorySizeMB\":128,\"maxMemoryUsedMB\":110},\"tracing\":{\"spanId\":\"cd47fc2b078a9d44\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 02:08:09.421",
        "@message": "{\"timestamp\": \"2026-10-02T02:08:09Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDenied) when calling the SendMessage operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-orders-exec/nightshift-orders is not authorized to perform: sqs:sendmessage on resource: arn:a...[1150 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 02:08:09.052",
        "@message": "{\"time\":\"2026-10-02T02:08:09.052Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"ae4ab5c0-6521-41ca-98d5-6fcfbc87ba41\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"26\",\"tracing\":{\"spanId\":\"cd47fc2b078a9d44\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 02:08:08.421",
        "@message": "{\"time\":\"2026-10-02T02:08:08.421Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"a254ff7e-556b-485f-945f-7325a575effd\",\"metrics\":{\"durationMs\":395.603,\"billedDurationMs\":396,\"memorySizeMB\":128,\"maxMemoryUsedMB\":110},\"tracing\":{\"spanId\":\"e73ecc059acf363b\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 595865,
    "scan_budget_left_bytes": 20335353
  }
}
  9. Step 9+180 sget_metricscited as evidence
    namespace=AWS/Lambda, statistic=Sum, period=60, metric=Errors, minutes=15
    Result (287 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "02:02:00Z",
        0
      ],
      [
        "02:03:00Z",
        0
      ],
      [
        "02:04:00Z",
        0
      ],
      [
        "02:05:00Z",
        0
      ],
      [
        "02:06:00Z",
        35
      ],
      [
        "02:07:00Z",
        43
      ]
    ],
    "summary": {
      "min": 0,
      "max": 43,
      "count": 6
    }
  }
}
  10. Step 10+182 snote_hypotheses
    hypotheses=ruled_out: bad_deploy ruled_out: config_regression ruled_out: slow_dependency ruled_out: timeout_regression ruled_out: throttling likely: iam_regression (AccessDenied when calling sqs:sendmessage due to permission changes)
    Result (31 characters)
    {
  "ok": "6 hypotheses recorded"
}
  11. Step 11+183 sfinish_investigation
    fault_category=iam_regression, evidence_steps=8,9, hypotheses=ruled_out: bad_deploy ruled_out: config_regression ruled_out: slow_dependency ruled_out: timeout_regression ruled_out: throttling likely: iam_regression, confidence=100, actions=, summary=The nightshift-orders Lambda function is failing with AccessDenied errors when trying to call the SQS SendMessage operation because the execution role lacks the required sqs:sendmessage permission., root_cause_component=orders, proposed_actions=Restore the missing sqs:sendmessage permission for the nightshift-orders execution role.
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation ae793b942101, gemini gemini-3.5-flash-lite, started 2026-10-02T02:08:06+00:00.

Answer

Root cause componentorders
Fault categoryiam_regression
Confidence100
Ended becausefinished

Summary

The nightshift-orders Lambda function is failing with AccessDenied errors when trying to call the SQS SendMessage operation because the execution role lacks the required sqs:sendmessage permission.

Impact

The page: nightshift-orders-errors was ALARM (since 2026-10-02T02:07:55Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [35.0 (02/10/26 02:06:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-02T02:08:07+00:00 step 1: get_alarm name=nightshift-orders-errors
  • 2026-10-02T02:08:07+00:00 step 2: list_recent_deployments service=orders (found nothing)
  • 2026-10-02T02:08:09+00:00 step 3: query_logs service=orders, query=fields @timestamp, level, message, correlation_id | filter level = "... (found nothing)
  • 2026-10-02T02:08:11+00:00 step 4: get_traces minutes=10, service=orders
  • 2026-10-02T02:08:11+00:00 step 5: get_function_config service=orders
  • 2026-10-02T02:08:12+00:00 step 6: lookup_recent_changes minutes=15
  • 2026-10-02T02:08:13+00:00 step 7: note_hypotheses hypotheses=ruled_out: bad_deploy (no recent deployments) ruled_out: config_regression (no ...
  • 2026-10-02T02:08:15+00:00 step 8: query_logs minutes=10, service=orders, query=fields @timestamp, @message | sort @timestamp desc | lim...
  • 2026-10-02T02:08:15+00:00 step 9: get_metrics namespace=AWS/Lambda, statistic=Sum, period=60, metric=Errors, minutes=15
  • 2026-10-02T02:08:17+00:00 step 10: note_hypotheses hypotheses=ruled_out: bad_deploy ruled_out: config_regression ruled_out: slow_dependency r...
  • 2026-10-02T02:08:18+00:00 step 11: finish_investigation fault_category=iam_regression, evidence_steps=8,9, hypotheses=ruled_out: bad_deploy ruled_...

Root cause evidence

  • Step 8, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-02 02:08:10.087","@message":"{"time":"2026-10-02T02:08:10.087Z","type":"platform.start","record":{"requestId":"05e2c27a-6402-4293-a6f3-c6b2e08cf88e","functionArn":"arn:aws...(older result, shortened)
  • Step 9, get_metrics: {"metric":"AWS/Lambda Errors {} Sum per 60s","window_minutes":15,"points":[["02:02:00Z",0.0],["02:03:00Z",0.0],["02:04:00Z",0.0],["02:05:00Z",0.0],["02:06:00Z",35.0],["02:07:00Z",43.0]],"summary":{"min":0.0,"max":43.0,"count":6}}

Proposed fix (for a human to decide; the agent changed nothing)

  • Restore the missing sqs:sendmessage permission for the nightshift-orders execution role.

Hypotheses

  • ruled_out: bad_deploy
  • ruled_out: config_regression
  • ruled_out: slow_dependency
  • ruled_out: timeout_regression
  • ruled_out: throttling
  • likely: iam_regression

Dropped along the way:

  • bad_deploy (no recent deployments)
  • config_regression (no recent config changes found in CloudTrail except log stream creation)
  • iam_regression (AccessDenied when calling sqs:sendmessage due to permission changes)
  • slow_dependency or dsql / timeout_regression or throttling (errors and high duration seen in X-Ray traces)

What the investigation cost

  • 5 model calls, 19,887 input and 553 output tokens (20,440 total)
  • 11 steps, 12 s of wall clock
  • 636,167 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments