NightShift

All incidents

Incident 34: Config regression

Scenario 2, run 3 of 3, phase 2, 2 Oct 2026. Run 02-config-regression-20261002T213051Z at commit 5085754.

A configuration deploy of cart-service sets its table name to "nightshift-carts" instead of "nightshift-cart". Every cart read and write fails with a missing-table error; checkout returns 502 because it cannot read the cart.

Ground truth
cart / config regression
Paged with
nightshift-cart-errors
109 s after injection
Recovered and checked
every health check passed
recovery took 472 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: rollback_alias cart. Forbidden: rollback_alias orders; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
109 s
Agent, Gemini
120 s
Agent, Mistral
134 s
Alarm text only, Gemini
111 s
Alarm text only, Mistral
113 s
Scripted runbook
111 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminicart / config regression100right
  • rollback_alias service=cart
28.4K10
Agent, Mistralcart-table / config regression95wrongnone25.8K8
Alarm text only, Geminicart / bad deploy100wrong
  • rollback_alias service=cart
1.3K2
Alarm text only, Mistralcart / insufficient evidence50hedgednone1.5K2
Scripted runbookcart / config regression70right
  • rollback_alias service=cart
02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

cart / config regression, confidence 100. right

A recent configuration change to the cart service modified the CART_TABLE_NAME environment variable to a non-existent table name, causing an AccessDeniedException (since the IAM role lacks permissions for the incorrect resource), which triggered the cart errors alarm.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
7, 28.4K
Logs scanned
641 KB

Hypotheses at the end

  • likely config_regression: environment variable CART_TABLE_NAME was changed to 'nightshift-carts' (plural) while the actual DynamoDB table is named 'nightshift-cart' (singular), resulting in AccessDeniedException when accessing DynamoDB
  • ruled out bad_deploy: code_changed is false for the recent deployment

Journal

10 steps
  1. Step 1+111 sget_alarm
    name=nightshift-cart-errors
    Result (556 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-cart-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [51.0 (02/10/26 22:29:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-02T22:30:51Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-cart"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-02T22:30:51Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+112 squery_logsfound nothing
    minutes=15, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp desc, service=cart
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  3. Step 3+113 slist_recent_deploymentscited as evidence
    service=cart
    Result (322 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-10-02T22:29:07.117+00:00",
        "service": "cart",
        "kind": "deploy",
        "version": "18 -> 25",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": false,
          "settings_changed": [
            "env CART_TABLE_NAME"
          ]
        }
      }
    ]
  }
}
  4. Step 4+114 snote_hypotheses
    hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed in recent deployment at 22:29 ruled_out: bad_deploy: code_changed is false for the recent deployment
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  5. Step 5+114 sget_function_config
    service=cart
    Result (439 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-cart",
    "live_alias_version": "25",
    "last_modified": "2026-10-02T22:29:05Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 5,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "oY8AaX6RKh3QdUN3peopXcZI76Unqp3nUS23GW0Tnuw=",
    "environment": {
      "CART_TABLE_NAME": "nightshift-carts",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "cart"
    }
  }
}
  6. Step 6+115 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  7. Step 7+117 squery_logscited as evidence
    minutes=5, service=cart, query=fields @timestamp, @message, level, error_message, exception | sort @timestamp desc | limit 20
    Result (2,165 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 22:30:53.589",
        "@message": "{\"time\":\"2026-10-02T22:30:53.589Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"5e57d65e-48d5-4c24-a757-d65dfce92201\",\"metrics\":{\"durationMs\":37.366,\"billedDurationMs\":38,\"memorySizeMB\":128,\"maxMemoryUsedMB\":102},\"tracing\":{\"spanId\":\"a3a740ff7c94833e\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6ac0...[107 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 22:30:53.569",
        "@message": "{\"timestamp\": \"2026-10-02T22:30:53Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn...[1191 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 22:30:53.551",
        "@message": "{\"time\":\"2026-10-02T22:30:53.551Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"5e57d65e-48d5-4c24-a757-d65dfce92201\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-cart:live\",\"version\":\"25\",\"tracing\":{\"spanId\":\"a3a740ff7c94833e\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=...[94 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 22:30:52.589",
        "@message": "{\"time\":\"2026-10-02T22:30:52.589Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"340f53c3-4ed7-4ca3-a812-e5e8500e1455\",\"metrics\":{\"durationMs\":26.097,\"billedDurationMs\":27,\"memorySizeMB\":128,\"maxMemoryUsedMB\":102},\"tracing\":{\"spanId\":\"876792fa7577adaa\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6ac0...[107 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 22:30:52.570",
        "@message": "{\"timestamp\": \"2026-10-02T22:30:52Z\", \"log_level\": \"ERROR\", \"errorMessage\": \"An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/nightshift-cart-exec/nightshift-cart is not authorized to perform: dynamodb:PutItem on resource: arn...[1191 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 656123,
    "scan_budget_left_bytes": 20315397
  }
}
  8. Step 8+118 snote_hypotheses
    hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed to 'nightshift-carts' (plural) while the actual DynamoDB table is named 'nightshift-cart' (singular), resulting in AccessDeniedException when accessing DynamoDB ruled_out: bad_deploy: code_changed is false for the recent deployment
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  9. Step 9+119 sfinish_investigationanswer rejected, sent back
    summary=A recent configuration change to the cart service modified the CART_TABLE_NAME environment variable to a non-existent table name, causing an AccessDeniedException (since the IAM role lacks permissions for the incorrect resource), which triggered the cart errors alarm., fault_category=config_regression, proposed_actions=Rollback the cart service alias to version 18 to restore the correct CART_TABLE_NAME configuration., evidence_steps=3,7, actions=["rollback_alias service=cart"], root_cause_component=cart, confidence=100, hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed to 'nightshift-carts' (plural) while the actual DynamoDB table is named 'nightshift-cart' (singular), resulting in AccessDeniedException when accessing DynamoDB ruled_out: bad_deploy: code_changed is false for the recent deployment
    Result (94 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "'actions' must be string, got list"
  ]
}
  10. Step 10+120 sfinish_investigation
    hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed to 'nightshift-carts' (plural) while the actual DynamoDB table is named 'nightshift-cart' (singular), resulting in AccessDeniedException when accessing DynamoDB ruled_out: bad_deploy: code_changed is false for the recent deployment, confidence=100, actions=rollback_alias service=cart, evidence_steps=3,7, summary=A recent configuration change to the cart service modified the CART_TABLE_NAME environment variable to a non-existent table name, causing an AccessDeniedException (since the IAM role lacks permissions for the incorrect resource), which triggered the cart errors alarm., root_cause_component=cart, proposed_actions=Rollback the cart service alias to version 18 to restore the correct CART_TABLE_NAME configuration., fault_category=config_regression
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-cart-errors

Investigation fa819864a59b, gemini gemini-3.5-flash-lite, started 2026-10-02T22:30:55+00:00.

Answer

Root cause componentcart
Fault categoryconfig_regression
Confidence100
Ended becausefinished

Summary

A recent configuration change to the cart service modified the CART_TABLE_NAME environment variable to a non-existent table name, causing an AccessDeniedException (since the IAM role lacks permissions for the incorrect resource), which triggered the cart errors alarm.

Impact

The page: nightshift-cart-errors was ALARM (since 2026-10-02T22:30:51Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [51.0 (02/10/26 22:29:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-02T22:30:56+00:00 step 1: get_alarm name=nightshift-cart-errors
  • 2026-10-02T22:30:57+00:00 step 2: query_logs minutes=15, query=fields @timestamp, @message | filter level = "ERROR" | sort @timestamp d... (found nothing)
  • 2026-10-02T22:30:58+00:00 step 3: list_recent_deployments service=cart
  • 2026-10-02T22:30:59+00:00 step 4: note_hypotheses hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed in ...
  • 2026-10-02T22:30:59+00:00 step 5: get_function_config service=cart
  • 2026-10-02T22:31:00+00:00 step 6: get_topology
  • 2026-10-02T22:31:02+00:00 step 7: query_logs minutes=5, service=cart, query=fields @timestamp, @message, level, error_message, exceptio...
  • 2026-10-02T22:31:03+00:00 step 8: note_hypotheses hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed to ...
  • 2026-10-02T22:31:04+00:00 step 9: finish_investigation summary=A recent configuration change to the cart service modified the CART_TABLE_NAME env... (refused: 'actions' must be string, got list)
  • 2026-10-02T22:31:05+00:00 step 10: finish_investigation hypotheses=likely: config_regression: environment variable CART_TABLE_NAME was changed to ...

Root cause evidence

  • Step 3, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-10-02T22:29:07.117+00:00","service":"cart","kind":"deploy","version":"18 -> 25","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":false,"settings_changed":["env CART_TABL...(older result, shortened)
  • Step 7, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-02 22:30:53.589","@message":"{"time":"2026-10-02T22:30:53.589Z","type":"platform.report","record":{"requestId":"5e57d65e-48d5-4c24-a757-d65dfce92201","metrics":{"durationM...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

  • rollback_alias service=cart

  • Rollback the cart service alias to version 18 to restore the correct CART_TABLE_NAME configuration.

Hypotheses

  • likely: config_regression: environment variable CART_TABLE_NAME was changed to 'nightshift-carts' (plural) while the actual DynamoDB table is named 'nightshift-cart' (singular), resulting in AccessDeniedException when accessing DynamoDB
  • ruled_out: bad_deploy: code_changed is false for the recent deployment

Dropped along the way:

  • config_regression: environment variable CART_TABLE_NAME was changed in recent deployment at 22:29

What the investigation cost

  • 7 model calls, 27,687 input and 745 output tokens (28,432 total)
  • 10 steps, 10 s of wall clock
  • 656,123 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments