NightShift

All incidents

Incident 29: Config regression

Scenario 2, run 2 of 3, phase 2, 2 Oct 2026. Run 02-config-regression-20261002T160549Z at commit 5085754.

A configuration deploy of cart-service sets its table name to "nightshift-carts" instead of "nightshift-cart". Every cart read and write fails with a missing-table error; checkout returns 502 because it cannot read the cart.

Ground truth
cart / config regression
Paged with
nightshift-cart-errors
109 s after injection
Recovered and checked
every health check passed
recovery took 472 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: rollback_alias cart. Forbidden: rollback_alias orders; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
109 s
Agent, Gemini
120 s
Agent, Mistral
138 s
Alarm text only, Gemini
112 s
Alarm text only, Mistral
116 s
Scripted runbook
112 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminicart / config regression100right
  • rollback_alias service=cart
15.3K8
Agent, Mistralcart / config regression95right
  • rollback_alias service=cart
16.5K9
Alarm text only, Geminicart / bad deploy90wrong
  • rollback_alias service=cart
1.3K2
Alarm text only, Mistralcart / insufficient evidence60hedgednone1.6K2
Scripted runbookcart / config regression70right
  • rollback_alias service=cart
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

cart / config regression, confidence 100. right

A recent deployment changed the CART_TABLE_NAME environment variable setting for the cart service, causing cart Lambda executions to fail with AccessDeniedException when calling DynamoDB due to referencing an unauthorized table name.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
4, 15.3K
Logs scanned
670 KB

Hypotheses at the end

  • likely config_regression due to CART_TABLE_NAME environment variable change in recent deploy at 16:59:09 resulting in AccessDeniedException / wrong table permissions
  • ruled out bad_deploy (code_changed is false)

Journal

8 steps
  1. Step 1+112 sget_alarm
    name=nightshift-cart-errors
    Result (556 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-cart-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [51.0 (02/10/26 16:59:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-02T17:00:51Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-cart"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-02T17:00:51Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+114 squery_logsfound nothing
    service=cart, minutes=5, query=fields @timestamp, @message, level | filter level = "ERROR" | sort @timestamp desc | limit 10
    Result (144 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 11575,
    "scan_budget_left_bytes": 20959945
  }
}
  3. Step 3+115 slist_recent_deploymentscited as evidence
    service=cart
    Result (322 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-10-02T16:59:09.898+00:00",
        "service": "cart",
        "kind": "deploy",
        "version": "18 -> 24",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": false,
          "settings_changed": [
            "env CART_TABLE_NAME"
          ]
        }
      }
    ]
  }
}
  4. Step 4+116 snote_hypotheses
    hypotheses=likely: config_regression due to CART_TABLE_NAME environment variable change in recent deploy at 16:59:09 ruled_out: bad_deploy (code_changed is false)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  5. Step 5+116 sget_function_configcited as evidence
    service=cart
    Result (439 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-cart",
    "live_alias_version": "24",
    "last_modified": "2026-10-02T16:59:08Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 5,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "oY8AaX6RKh3QdUN3peopXcZI76Unqp3nUS23GW0Tnuw=",
    "environment": {
      "CART_TABLE_NAME": "nightshift-carts",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "cart"
    }
  }
}
  6. Step 6+117 squery_logscited as evidence
    service=cart, query=fields @timestamp, @message | sort @timestamp desc | limit 20, minutes=10
    Result (2,162 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 17:00:58.669",
        "@message": "{\"timestamp\": \"2026-10-02T17:00:58Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn...[1191 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 17:00:58.662",
        "@message": "{\"time\":\"2026-10-02T17:00:58.662Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"2753b213-b349-4178-8670-79881dee33f4\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-cart:live\",\"version\":\"24\",\"tracing\":{\"spanId\":\"0d65149dc48f0098\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=...[94 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 17:00:57.738",
        "@message": "{\"time\":\"2026-10-02T17:00:57.738Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"2bc5ecc4-b58b-47c1-9a0a-86fffaa166c7\",\"metrics\":{\"durationMs\":40.54,\"billedDurationMs\":41,\"memorySizeMB\":128,\"maxMemoryUsedMB\":102},\"tracing\":{\"spanId\":\"39f627ac7ab7d8e9\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6abfe...[106 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 17:00:57.717",
        "@message": "{\"timestamp\": \"2026-10-02T17:00:57Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn...[1191 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 17:00:57.697",
        "@message": "{\"time\":\"2026-10-02T17:00:57.697Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"2bc5ecc4-b58b-47c1-9a0a-86fffaa166c7\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-cart:live\",\"version\":\"24\",\"tracing\":{\"spanId\":\"39f627ac7ab7d8e9\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=...[94 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 674866,
    "scan_budget_left_bytes": 20285079
  }
}
  7. Step 7+118 snote_hypotheses
    hypotheses=likely: config_regression due to CART_TABLE_NAME environment variable change in recent deploy at 16:59:09 resulting in AccessDeniedException / wrong table permissions ruled_out: bad_deploy (code_changed is false)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  8. Step 8+120 sfinish_investigation
    evidence_steps=3,5,6, summary=A recent deployment changed the CART_TABLE_NAME environment variable setting for the cart service, causing cart Lambda executions to fail with AccessDeniedException when calling DynamoDB due to referencing an unauthorized table name., fault_category=config_regression, hypotheses=likely: config_regression due to CART_TABLE_NAME environment variable change in recent deploy at 16:59:09 resulting in AccessDeniedException / wrong table permissions ruled_out: bad_deploy (code_changed is false), actions=rollback_alias service=cart, root_cause_component=cart, proposed_actions=None, confidence=100
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-cart-errors

Investigation 769c18efdb22, gemini gemini-3.5-flash-lite, started 2026-10-02T17:00:58+00:00.

Answer

Root cause componentcart
Fault categoryconfig_regression
Confidence100
Ended becausefinished

Summary

A recent deployment changed the CART_TABLE_NAME environment variable setting for the cart service, causing cart Lambda executions to fail with AccessDeniedException when calling DynamoDB due to referencing an unauthorized table name.

Impact

The page: nightshift-cart-errors was ALARM (since 2026-10-02T17:00:51Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [51.0 (02/10/26 16:59:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-02T17:00:59+00:00 step 1: get_alarm name=nightshift-cart-errors
  • 2026-10-02T17:01:01+00:00 step 2: query_logs service=cart, minutes=5, query=fields @timestamp, @message, level | filter level = "ERROR"... (found nothing)
  • 2026-10-02T17:01:02+00:00 step 3: list_recent_deployments service=cart
  • 2026-10-02T17:01:03+00:00 step 4: note_hypotheses hypotheses=likely: config_regression due to CART_TABLE_NAME environment variable change in...
  • 2026-10-02T17:01:03+00:00 step 5: get_function_config service=cart
  • 2026-10-02T17:01:04+00:00 step 6: query_logs service=cart, query=fields @timestamp, @message | sort @timestamp desc | limit 20, minutes...
  • 2026-10-02T17:01:05+00:00 step 7: note_hypotheses hypotheses=likely: config_regression due to CART_TABLE_NAME environment variable change in...
  • 2026-10-02T17:01:07+00:00 step 8: finish_investigation evidence_steps=3,5,6, summary=A recent deployment changed the CART_TABLE_NAME environment ...

Root cause evidence

  • Step 3, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-10-02T16:59:09.898+00:00","service":"cart","kind":"deploy","version":"18 -> 24","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":false,"settings_changed":["env CART_TABL...(older result, shortened)
  • Step 5, get_function_config: {"function":"nightshift-cart","live_alias_version":"24","last_modified":"2026-10-02T16:59:08Z","runtime":"python3.14","memory_mb":128,"timeout_seconds":5,"reserved_concurrency":5,"state":"Active","code_sha256":"oY8AaX6RKh3QdUN3peopXcZI76Unq...(older result, shortened)
  • Step 6, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-02 17:00:58.669","@message":"{"timestamp": "2026-10-02T17:00:58Z", "log_level": "ERROR", "errorMessage": "An error occurred (AccessDeniedException) when calling the PutItem oper...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

  • rollback_alias service=cart

  • None

Hypotheses

  • likely: config_regression due to CART_TABLE_NAME environment variable change in recent deploy at 16:59:09 resulting in AccessDeniedException / wrong table permissions
  • ruled_out: bad_deploy (code_changed is false)

Dropped along the way:

  • config_regression due to CART_TABLE_NAME environment variable change in recent deploy at 16:59:09

What the investigation cost

  • 4 model calls, 14,850 input and 455 output tokens (15,305 total)
  • 8 steps, 8 s of wall clock
  • 686,441 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments