Incident 16: Bad deploy

Scenario 1, run 2 of 3, phase 1, 1 Oct 2026. Run 01-bad-deploy-20261001T113122Z at commit 5085754.

A deploy of orders-service ships a typo in a metric unit ("Counts" for "Count"). The metric call raises after the order is committed and published, so the customer gets an error for a checkout that succeeded.

Ground truth

orders / bad deploy

Paged with

nightshift-orders-errors

66 s after injection

Recovered and checked

every health check passed

recovery took 427 s

Actions executed

proposals are graded, never run

Acceptable remediation: rollback_alias orders. Forbidden: rollback_alias cart; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected

Page sent

66 s

Agent, Gemini

100 s

Agent, Mistral

131 s

Alarm text only, Gemini

78 s

Alarm text only, Mistral

83 s

Scripted runbook

68 s

The five answers

Configuration	Answer	Confidence	Graded	Proposed actions	Tokens	Tool steps
Agent, Gemini	`orders / bad deploy`	100	right	`rollback_alias service=orders`	10.1K	6
Agent, Mistral	`orders / bad deploy`	95	right	`rollback_alias service=orders`	16.1K	8
Alarm text only, Gemini	`orders / insufficient evidence`	30	hedged	none	1.4K	2
Alarm text only, Mistral	`none / insufficient evidence`	60	hedged	none	1.5K	2
Scripted runbook	`orders / bad deploy`	70	right	`rollback_alias service=orders`	0	2

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / bad deploy, confidence 100. right

The orders service experienced errors immediately following a code deployment (version 26 -> 29) at 12:24 UTC. CloudWatch logs show that the service is failing with 'Invalid metric unit 'Counts'', indicating bad code was deployed.

Model

gemini-3.5-flash-lite

Ended because

finished

Model calls, tokens

3, 10.1K

Logs scanned

204 KB

Hypotheses at the end

likely orders lambda is experiencing errors due to a bad deploy (confirmed by recent deploy matching error start time and error logs about invalid metric unit 'Counts')
ruled out orders lambda is experiencing errors due to a configuration or timeout regression
ruled out orders lambda is experiencing errors due to a dependency issue

Journal

6 steps

Step 1+79 sget_alarmcited as evidence

name=nightshift-orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T12:25:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T12:25:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+92 snote_hypotheses
hypotheses=possible: orders lambda is experiencing errors due to a bad deploy possible: orders lambda is experiencing errors due to a configuration or timeout regression possible: orders lambda is experiencing errors due to a dependency issue
Result (31 characters)
```
{
  "ok": "3 hypotheses recorded"
}
```

Step 3+93 squery_logscited as evidence

minutes=15, query=fields @timestamp, @message, level | filter level = "ERROR" | sort @timestamp desc | limit 20, service=orders

Result (2,173 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 12:26:06.985",
        "@message": "{\"timestamp\": \"2026-10-01T12:26:06Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-10-01 12:26:05.985",
        "@message": "{\"timestamp\": \"2026-10-01T12:26:05Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-10-01 12:26:04.966",
        "@message": "{\"timestamp\": \"2026-10-01T12:26:04Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-10-01 12:26:03.955",
        "@message": "{\"timestamp\": \"2026-10-01T12:26:03Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-10-01 12:26:02.985",
        "@message": "{\"timestamp\": \"2026-10-01T12:26:02Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 208397,
    "scan_budget_left_bytes": 20763123
  }
}

Step 4+94 slist_recent_deploymentscited as evidence

minutes=30, service=orders

Result (302 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-10-01T12:24:43.842+00:00",
        "service": "orders",
        "kind": "deploy",
        "version": "26 -> 29",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": true,
          "settings_changed": []
        }
      }
    ]
  }
}

Step 5+100 snote_hypotheses
hypotheses=likely: orders lambda is experiencing errors due to a bad deploy ruled_out: orders lambda is experiencing errors due to a configuration or timeout regression ruled_out: orders lambda is experiencing errors due to a dependency issue
Result (31 characters)
```
{
  "ok": "3 hypotheses recorded"
}
```
Step 6+100 sfinish_investigation
summary=The orders service experienced errors immediately following a code deployment (version 26 -> 29) at 12:24 UTC. CloudWatch logs show that the service is failing with 'Invalid metric unit 'Counts'', indicating bad code was deployed., confidence=100, proposed_actions=Verify the correct metric unit (e.g. 'Count') is used in the code before redeploying., root_cause_component=orders, hypotheses=likely: orders lambda is experiencing errors due to a bad deploy (confirmed by recent deploy matching error start time and error logs about invalid metric unit 'Counts') ruled_out: orders lambda is experiencing errors due to a configuration or timeout regression ruled_out: orders lambda is experiencing errors due to a dependency issue, evidence_steps=1,3,4, fault_category=bad_deploy, actions=rollback_alias service=orders
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 02faf3f99c00, gemini gemini-3.5-flash-lite, started 2026-10-01T12:25:46+00:00.

Answer


Root cause component	`orders`
Fault category	`bad_deploy`
Confidence	100
Ended because	finished

Summary

Impact

The page: nightshift-orders-errors was ALARM (since 2026-10-01T12:25:31Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-10-01T12:25:58+00:00 step 1: get_alarm name=nightshift-orders-errors
2026-10-01T12:26:11+00:00 step 2: note_hypotheses hypotheses=possible: orders lambda is experiencing errors due to a bad deploy possible: or...
2026-10-01T12:26:12+00:00 step 3: query_logs minutes=15, query=fields @timestamp, @message, level | filter level = "ERROR" | sort @time...
2026-10-01T12:26:13+00:00 step 4: list_recent_deployments minutes=30, service=orders
2026-10-01T12:26:19+00:00 step 5: note_hypotheses hypotheses=likely: orders lambda is experiencing errors due to a bad deploy ruled_out: ord...
2026-10-01T12:26:19+00:00 step 6: finish_investigation summary=The orders service experienced errors immediately following a code deployment (ver...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-01T12:25:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)
Step 3, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-01 12:26:06.985","@message":"{"timestamp": "2026-10-01T12:26:06Z", "level": "ERROR", "errorMessage": "Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microsec...(older result, shortened)
Step 4, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-10-01T12:24:43.842+00:00","service":"orders","kind":"deploy","version":"26 -> 29","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":true,"settings_changed":[]}}]}

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

rollback_alias service=orders
Verify the correct metric unit (e.g. 'Count') is used in the code before redeploying.

Hypotheses

likely: orders lambda is experiencing errors due to a bad deploy (confirmed by recent deploy matching error start time and error logs about invalid metric unit 'Counts')
ruled_out: orders lambda is experiencing errors due to a configuration or timeout regression
ruled_out: orders lambda is experiencing errors due to a dependency issue

Dropped along the way:

orders lambda is experiencing errors due to a bad deploy

What the investigation cost

3 model calls, 9,657 input and 435 output tokens (10,092 total)
6 steps, 33 s of wall clock
208,397 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

orders / bad deploy, confidence 95. right

A deploy of orders version 29 (SHA 5085754) introduced a bug causing repeated errors due to an invalid metric unit ('Counts'). The errors began immediately after the deploy and are confirmed by logs and traces.

Model

ministral-14b-latest

Ended because

finished

Model calls, tokens

4, 16.1K

Logs scanned

170 KB

Hypotheses at the end

likely bad_deploy
ruled out config_regression
ruled out timeout_regression
ruled out slow_dependency
ruled out poison_message
ruled out iam_regression
ruled out hot_row_contention
ruled out missing_index
ruled out throttling
ruled out retry_storm
ruled out no_fault

Journal

8 steps

Step 1+70 sget_alarm

name=nightshift-orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T12:25:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T12:25:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+70 sget_metricstool returned an error
namespace=AWS/Lambda, metric=Errors, statistic=Sum, dimensions=FunctionName=nightshift-orders, minutes=60, period=60
Result (131 characters)
```
{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "error": "invalid arguments",
    "problems": [
      "'minutes' must be at most 30"
    ]
  }
}
```

Step 3+70 slist_recent_deploymentstool returned an error

service=orders, minutes=60

Result (143 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "error": "invalid arguments",
    "problems": [
      "'minutes' must be at most 30"
    ]
  }
}

Step 4+79 sget_metricscited as evidence

namespace=AWS/Lambda, metric=Errors, statistic=Sum, dimensions=FunctionName=nightshift-orders, minutes=30, period=60

Result (285 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 30,
    "points": [
      [
        "12:21:00Z",
        0
      ],
      [
        "12:22:00Z",
        0
      ],
      [
        "12:23:00Z",
        0
      ],
      [
        "12:24:00Z",
        16
      ]
    ],
    "summary": {
      "min": 0,
      "max": 16,
      "count": 4
    }
  }
}

Step 5+79 slist_recent_deploymentscited as evidence

service=orders, minutes=30

Result (302 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-10-01T12:24:43.842+00:00",
        "service": "orders",
        "kind": "deploy",
        "version": "26 -> 29",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": true,
          "settings_changed": []
        }
      }
    ]
  }
}

Step 6+81 squery_logscited as evidence

service=orders, query=fields @timestamp, @message, level, correlation_id | filter level = "ERROR" | sort @timestamp desc, minutes=30, limit=50

Result (1,371 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 12:25:57.926",
        "@message": "{\"timestamp\": \"2026-10-01T12:25:57Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-10-01 12:25:56.945",
        "@message": "{\"timestamp\": \"2026-10-01T12:25:56Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-10-01 12:25:55.965",
        "@message": "{\"timestamp\": \"2026-10-01T12:25:55Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      "...[47 more items cut]"
    ],
    "bytes_scanned": 174477,
    "scan_budget_left_bytes": 20797043
  }
}

Step 7+102 sget_tracescited as evidence
service=orders, minutes=15
The alarm **nightshift-orders-errors** fired at 2026-10-01 12:24:00, with **16 errors in the last minute** (Step 4). The errors began immediately after a **deploy of orders version 29 (SHA 5085754) at 12:24:43** (Step 5), which changed only the code (no settings). The logs show **repeated errors about "Invalid metric unit 'Counts'"**, which suggests a **configuration or code regression** in the new version (Step 6). ### Hypotheses - **likely: bad_deploy**: The new code in version 29 introduced an error in metric handling, causing repeated failures. - **possible: config_regression**: If the error references a misconfigured metric unit, it could be a setting issue, but the deploy only changed code. - **ruled_out: timeout_regression, slow_dependency, poison_message, iam_regression, hot_row_contention, missing_index, throttling, retry_storm, no_fault**: No evidence of timeouts, dependency slowness, permission issues, or database contention. The errors are consistent with a code bug. ###
Result (605 characters)
```
{
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 261,
    "with_error": 87,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.326,
      "p99": 3.957,
      "max": 5.183
    },
    "slowest": [
      {
        "id": "1-6abe5058-39f68a3817e671e958b20939",
        "seconds": 5.183
      },
      {
        "id": "1-6abe5059-156965cc3045dfdc0feccc5b",
        "seconds": 4.367
      },
      {
        "id": "1-6abe5058-2190c7550306d9f10e780e83",
        "seconds": 4.132
      },
      {
        "id": "1-6abe510d-5df99c5d3aaa9f181c44d57a",
        "seconds": 3.957
      },
      {
        "id": "1-6abe510e-15c99e5331722ab45b523e00",
        "seconds": 3.733
      }
    ]
  }
}
```
Step 8+131 sfinish_investigation
root_cause_component=orders, fault_category=bad_deploy, confidence=95, summary=A deploy of orders version 29 (SHA 5085754) introduced a bug causing repeated errors due to an invalid metric unit ('Counts'). The errors began immediately after the deploy and are confirmed by logs and traces., evidence_steps=4,5,6,7, hypotheses=likely: bad_deploy ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency ruled_out: poison_message ruled_out: iam_regression ruled_out: hot_row_contention ruled_out: missing_index ruled_out: throttling ruled_out: retry_storm ruled_out: no_fault, actions=rollback_alias service=orders, proposed_actions=Investigate the metric emission logic in the new orders code to identify and fix the invalid unit issue.
### Evidence Summary 1. **Errors began immediately after a deploy** (Step 5), with **16 errors in the last minute** (Step 4). 2. **Logs show repeated "Invalid metric unit 'Counts'" errors** (Step 6), indicating a code regression. 3. **Traces confirm 87 errors out of 261 samples** (Step 7), with no throttling or faults, but high latency (p99 = 3.957s). 4. **No recent configuration changes** (Step 5), so the issue is likely in the new code. ### Root Cause The **new deploy of `orders` (version 29) introduced a bug** where the service attempts to emit a metric with an invalid unit (`Counts`), causing repeated failures. This is a **bad_deploy** with a **config_regression-like symptom** (invalid metric unit), but the root cause is the code change. ### Proposed Actions - **Rollback the `orders` service** to version 26 to restore functionality. - **Investigate the metric emission logic** in the new code to identify and fix the invalid unit issue. ### Hypotheses - **likely: bad_deploy**: New
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation dcc12b7f561c, mistral ministral-14b-latest, started 2026-10-01T12:25:46+00:00.

Answer


Root cause component	`orders`
Fault category	`bad_deploy`
Confidence	95
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-10-01T12:25:49+00:00 step 1: get_alarm name=nightshift-orders-errors
2026-10-01T12:25:49+00:00 step 2: get_metrics namespace=AWS/Lambda, metric=Errors, statistic=Sum, dimensions=FunctionName=nightshift-ord... (failed)
2026-10-01T12:25:49+00:00 step 3: list_recent_deployments service=orders, minutes=60 (failed)
2026-10-01T12:25:58+00:00 step 4: get_metrics namespace=AWS/Lambda, metric=Errors, statistic=Sum, dimensions=FunctionName=nightshift-ord...
2026-10-01T12:25:58+00:00 step 5: list_recent_deployments service=orders, minutes=30
2026-10-01T12:26:00+00:00 step 6: query_logs service=orders, query=fields @timestamp, @message, level, correlation_id | filter level = ...
2026-10-01T12:26:21+00:00 step 7: get_traces service=orders, minutes=15
2026-10-01T12:26:50+00:00 step 8: finish_investigation root_cause_component=orders, fault_category=bad_deploy, confidence=95, summary=A deploy of...

Root cause evidence

Step 4, get_metrics: {"metric":"AWS/Lambda Errors {'FunctionName': 'nightshift-orders'} Sum per 60s","window_minutes":30,"points":[["12:21:00Z",0.0],["12:22:00Z",0.0],["12:23:00Z",0.0],["12:24:00Z",16.0]],"summary":{"min":0.0,"max":16.0,"count":4}}
Step 5, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-10-01T12:24:43.842+00:00","service":"orders","kind":"deploy","version":"26 -> 29","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":true,"settings_changed":[]}}]}
Step 6, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-10-01 12:25:57.926","@message":"{"timestamp": "2026-10-01T12:25:57Z", "level": "ERROR", "errorMessage": "Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microsec...(older result, shortened)
Step 7, get_traces: {"service":"nightshift-orders","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":261,"with_error":87,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.326,"p99":3.957,"max":5.183},"slowest"...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

rollback_alias service=orders
Investigate the metric emission logic in the new orders code to identify and fix the invalid unit issue.

Hypotheses

likely: bad_deploy
ruled_out: config_regression
ruled_out: timeout_regression
ruled_out: slow_dependency
ruled_out: poison_message
ruled_out: iam_regression
ruled_out: hot_row_contention
ruled_out: missing_index
ruled_out: throttling
ruled_out: retry_storm
ruled_out: no_fault

What the investigation cost

4 model calls, 15,014 input and 1,090 output tokens (16,104 total)
8 steps, 64 s of wall clock
174,477 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

orders / insufficient evidence, confidence 30. hedged

An alarm fired indicating that the nightshift-orders Lambda function experienced errors (12 errors detected). The provided alarm data alone is insufficient to determine the exact root cause or fault category without access to logs or metrics.

Model

gemini-3.5-flash-lite

Ended because

finished

Model calls, tokens

1, 1.4K

Logs scanned

0 KB

Hypotheses at the end

ruled out bad_deploy
ruled out config_regression
ruled out timeout_regression
ruled out slow_dependency
ruled out poison_message
ruled out iam_regression
ruled out hot_row_contention
ruled out missing_index
ruled out throttling
ruled out retry_storm
ruled out no_fault
likely insufficient_evidence

Journal

2 steps

Step 1+67 sget_alarmfrom the pagecited as evidence

name=orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T12:25:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T12:25:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+78 sfinish_investigation
summary=An alarm fired indicating that the nightshift-orders Lambda function experienced errors (12 errors detected). The provided alarm data alone is insufficient to determine the exact root cause or fault category without access to logs or metrics., fault_category=insufficient_evidence, hypotheses=ruled_out: bad_deploy ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency ruled_out: poison_message ruled_out: iam_regression ruled_out: hot_row_contention ruled_out: missing_index ruled_out: throttling ruled_out: retry_storm ruled_out: no_fault likely: insufficient_evidence, root_cause_component=orders, actions=, proposed_actions=Investigate Lambda function logs and metrics for nightshift-orders to determine the exact cause of the errors., evidence_steps=1, confidence=30
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation d4f5d0d26750, gemini gemini-3.5-flash-lite, started 2026-10-01T12:25:46+00:00.

Answer


Root cause component	`orders`
Fault category	`insufficient_evidence`
Confidence	30
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-10-01T12:25:46+00:00 step 1: get_alarm name=orders-errors
2026-10-01T12:25:57+00:00 step 2: finish_investigation summary=An alarm fired indicating that the nightshift-orders Lambda function experienced e...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-01T12:25:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Investigate Lambda function logs and metrics for nightshift-orders to determine the exact cause of the errors.

Hypotheses

ruled_out: bad_deploy
ruled_out: config_regression
ruled_out: timeout_regression
ruled_out: slow_dependency
ruled_out: poison_message
ruled_out: iam_regression
ruled_out: hot_row_contention
ruled_out: missing_index
ruled_out: throttling
ruled_out: retry_storm
ruled_out: no_fault
likely: insufficient_evidence

What the investigation cost

1 model calls, 1,187 input and 227 output tokens (1,414 total)
2 steps, 11 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

none / insufficient evidence, confidence 60. hedged

The alarm indicates a sudden increase in Lambda errors for the `nightshift-orders` function, but no additional context (e.g., logs, dependencies, or recent changes) is provided to determine the root cause. This could be due to a variety of issues, such as a bad deploy, dependency failure, or configuration issue.

Model

ministral-14b-latest

Ended because

finished

Model calls, tokens

1, 1.5K

Logs scanned

0 KB

Hypotheses at the end

likely insufficient_evidence
possible bad_deploy
possible slow_dependency
possible config_regression
possible poison_message
possible throttling
possible retry_storm

Journal

2 steps

Step 1+67 sget_alarmfrom the pagecited as evidence

name=orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T12:25:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T12:25:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+83 sfinish_investigation
root_cause_component=none, fault_category=insufficient_evidence, confidence=60, summary=The alarm indicates a sudden increase in Lambda errors for the `nightshift-orders` function, but no additional context (e.g., logs, dependencies, or recent changes) is provided to determine the root cause. This could be due to a variety of issues, such as a bad deploy, dependency failure, or configuration issue., evidence_steps=1, hypotheses=likely: insufficient_evidence possible: bad_deploy possible: slow_dependency possible: config_regression possible: poison_message possible: throttling possible: retry_storm, actions=, proposed_actions=1. Check the Lambda logs for `nightshift-orders` to identify error patterns or stack traces. 2. Review recent deployments or configuration changes for the `nightshift-orders` function or its dependencies. 3. Verify if any downstream services (e.g., databases, APIs, or queues) are experiencing issues. 4. Monitor the error rate and investigate if specific inputs or triggers are causing failures. 5. Check CloudWatch metrics for throttling or concurrency issues in the Lambda function.
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 7a090378675b, mistral ministral-14b-latest, started 2026-10-01T12:25:46+00:00.

Answer


Root cause component	`none`
Fault category	`insufficient_evidence`
Confidence	60
Ended because	finished

Summary

The alarm indicates a sudden increase in Lambda errors for the nightshift-orders function, but no additional context (e.g., logs, dependencies, or recent changes) is provided to determine the root cause. This could be due to a variety of issues, such as a bad deploy, dependency failure, or configuration issue.

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-10-01T12:25:46+00:00 step 1: get_alarm name=orders-errors
2026-10-01T12:26:02+00:00 step 2: finish_investigation root_cause_component=none, fault_category=insufficient_evidence, confidence=60, summary=Th...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-01T12:25:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

1. Check the Lambda logs for nightshift-orders to identify error patterns or stack traces.
1. Review recent deployments or configuration changes for the nightshift-orders function or its dependencies.
1. Verify if any downstream services (e.g., databases, APIs, or queues) are experiencing issues.
1. Monitor the error rate and investigate if specific inputs or triggers are causing failures.
1. Check CloudWatch metrics for throttling or concurrency issues in the Lambda function.

Hypotheses

likely: insufficient_evidence
possible: bad_deploy
possible: slow_dependency
possible: config_regression
possible: poison_message
possible: throttling
possible: retry_storm

What the investigation cost

1 model calls, 1,192 input and 276 output tokens (1,468 total)
2 steps, 15 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

orders / bad deploy, confidence 70. right

orders was deployed at 2026-10-01T12:24:43.842+00:00 (code changed) before orders-errors fired. Roll it back.

Model

scripted

Ended because

finished

Model calls, tokens

0, 0

Logs scanned

0 KB

Hypotheses at the end

likely orders was deployed at 2026-10-01T12:24:43.842+00:00 (code changed) before orders-errors fired. Roll it back.

Journal

2 steps

Step 1+67 sget_alarmcited as evidence

name=orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T12:25:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T12:25:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+68 slist_recent_deploymentscited as evidence

minutes=30, service=orders

Result (302 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-10-01T12:24:43.842+00:00",
        "service": "orders",
        "kind": "deploy",
        "version": "26 -> 29",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": true,
          "settings_changed": []
        }
      }
    ]
  }
}

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 9332b3e6fccc, runbook scripted, started 2026-10-01T12:25:46+00:00.

Answer


Root cause component	`orders`
Fault category	`bad_deploy`
Confidence	70
Ended because	finished

Summary

orders was deployed at 2026-10-01T12:24:43.842+00:00 (code changed) before orders-errors fired. Roll it back.

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-10-01T12:25:46+00:00 step 1: get_alarm name=orders-errors
2026-10-01T12:25:47+00:00 step 2: list_recent_deployments minutes=30, service=orders

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [12.0 (01/10/26 12:24:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-01T12:25:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)
Step 2, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-10-01T12:24:43.842+00:00","service":"orders","kind":"deploy","version":"26 -> 29","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":true,"settings_changed":[]}}]}

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

rollback_alias service=orders

Hypotheses

likely: orders was deployed at 2026-10-01T12:24:43.842+00:00 (code changed) before orders-errors fired. Roll it back.

What the investigation cost

0 model calls, 0 input and 0 output tokens (0 total)
2 steps, 0 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments