Incident 2: Bad deploy

Scenario 1, run 3 of 3, phase 1, 30 Sep 2026. Run 01-bad-deploy-20260930T155157Z at commit 5085754.

A deploy of orders-service ships a typo in a metric unit ("Counts" for "Count"). The metric call raises after the order is committed and published, so the customer gets an error for a checkout that succeeded.

Ground truth

orders / bad deploy

Paged with

nightshift-orders-errors

88 s after injection

Recovered and checked

every health check passed

recovery took 452 s

Actions executed

proposals are graded, never run

Acceptable remediation: rollback_alias orders. Forbidden: rollback_alias cart; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected

Page sent

88 s

Agent, Gemini

101 s

Agent, Mistral

130 s

Alarm text only, Gemini

95 s

Alarm text only, Mistral

96 s

Scripted runbook

91 s

The five answers

Configuration	Answer	Confidence	Graded	Proposed actions	Tokens	Tool steps
Agent, Gemini	`orders / bad deploy`	100	right	`rollback_alias service=orders`	14.9K	6
Agent, Mistral	`orders / bad deploy`	95	right	`rollback_alias service=orders`	26.6K	9
Alarm text only, Gemini	`orders / insufficient evidence`	50	hedged	none	1.4K	2
Alarm text only, Mistral	`none / insufficient evidence`	95	hedged	none	1.6K	2
Scripted runbook	`orders / bad deploy`	70	right	`rollback_alias service=orders`	0	2

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / bad deploy, confidence 100. right

A recent deployment (version 27, commit 5085754) introduced code that passes an invalid metric unit 'Counts' to CloudWatch, causing the orders Lambda function to throw errors and triggering the alarm.

Model

gemini-3.5-flash-lite

Ended because

finished

Model calls, tokens

4, 14.9K

Logs scanned

217 KB

Hypotheses at the end

likely bad_deploy caused the orders function to fail due to invalid metric unit 'Counts'
ruled out config_regression
ruled out timeout_regression
ruled out slow_dependency
ruled out throttling
ruled out retry_storm

Journal

6 steps

Step 1+91 sget_alarm

name=nightshift-orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T16:42:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T16:42:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+93 squery_logscited as evidence

service=orders, minutes=5, query=fields @timestamp, @message, level, reason | filter level = "ERROR" | sort @timestamp desc | limit 20

Result (2,173 characters)

{
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-09-30 16:42:29.951",
        "@message": "{\"timestamp\": \"2026-09-30T16:42:29Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 16:42:29.311",
        "@message": "{\"timestamp\": \"2026-09-30T16:42:29Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 16:42:28.111",
        "@message": "{\"timestamp\": \"2026-09-30T16:42:28Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 16:42:27.193",
        "@message": "{\"timestamp\": \"2026-09-30T16:42:27Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      {
        "@timestamp": "2026-09-30 16:42:26.311",
        "@message": "{\"timestamp\": \"2026-09-30T16:42:26Z\", \"level\": \"ERROR\", \"errorMessage\": \"Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microseconds', 'Milliseconds', 'Bytes', 'Kilobytes', 'Megabytes', 'Gigabytes', 'Terabytes', 'Bits', 'Kilobits', 'Megabits', 'Gigabits', 'Terabits', 'Percent', '...[1140 chars cut]",
        "level": "ERROR"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 222096,
    "scan_budget_left_bytes": 20749424
  }
}

Step 3+94 slist_recent_deploymentscited as evidence

minutes=15, service=orders

Result (302 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 15,
    "moves": [
      {
        "at": "2026-09-30T16:41:10.092+00:00",
        "service": "orders",
        "kind": "deploy",
        "version": "26 -> 27",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": true,
          "settings_changed": []
        }
      }
    ]
  }
}

Step 4+95 snote_hypotheses
hypotheses=likely: bad_deploy caused the orders function to fail due to invalid metric unit 'Counts' ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency ruled_out: throttling ruled_out: retry_storm
Result (31 characters)
```
{
  "ok": "6 hypotheses recorded"
}
```
Step 5+100 sfinish_investigationanswer rejected, sent back
summary=A recent deployment (version 27, commit 5085754) introduced code that passes an invalid metric unit 'Counts' to CloudWatch, causing the orders Lambda function to throw errors and triggering the alarm., evidence_steps=2,3, actions=["rollback_alias service=orders"], proposed_actions=Verify code changes in git commit 5085754 for orders service regarding metric unit 'Counts' before redeploying., confidence=100, fault_category=bad_deploy, root_cause_component=orders, hypotheses=likely: bad_deploy caused the orders function to fail due to invalid metric unit 'Counts' ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency ruled_out: throttling ruled_out: retry_storm
Result (94 characters)
```
{
  "error": "finish_investigation rejected",
  "problems": [
    "'actions' must be string, got list"
  ]
}
```
Step 6+101 sfinish_investigation
confidence=100, proposed_actions=Verify code changes in git commit 5085754 for orders service regarding metric unit 'Counts' before redeploying., root_cause_component=orders, summary=A recent deployment (version 27, commit 5085754) introduced code that passes an invalid metric unit 'Counts' to CloudWatch, causing the orders Lambda function to throw errors and triggering the alarm., hypotheses=likely: bad_deploy caused the orders function to fail due to invalid metric unit 'Counts' ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency ruled_out: throttling ruled_out: retry_storm, fault_category=bad_deploy, actions=rollback_alias service=orders, evidence_steps=2,3
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 7b5dda1af670, gemini gemini-3.5-flash-lite, started 2026-09-30T16:42:37+00:00.

Answer


Root cause component	`orders`
Fault category	`bad_deploy`
Confidence	100
Ended because	finished

Summary

Impact

The page: nightshift-orders-errors was ALARM (since 2026-09-30T16:42:31Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T16:42:38+00:00 step 1: get_alarm name=nightshift-orders-errors
2026-09-30T16:42:40+00:00 step 2: query_logs service=orders, minutes=5, query=fields @timestamp, @message, level, reason | filter level...
2026-09-30T16:42:41+00:00 step 3: list_recent_deployments minutes=15, service=orders
2026-09-30T16:42:42+00:00 step 4: note_hypotheses hypotheses=likely: bad_deploy caused the orders function to fail due to invalid metric uni...
2026-09-30T16:42:47+00:00 step 5: finish_investigation summary=A recent deployment (version 27, commit 5085754) introduced code that passes an in... (refused: 'actions' must be string, got list)
2026-09-30T16:42:48+00:00 step 6: finish_investigation confidence=100, proposed_actions=Verify code changes in git commit 5085754 for orders serv...

Root cause evidence

Step 2, query_logs: {"status":"Complete","rows":[{"@timestamp":"2026-09-30 16:42:29.951","@message":"{"timestamp": "2026-09-30T16:42:29Z", "level": "ERROR", "errorMessage": "Invalid metric unit 'Counts', expected either option: ['Seconds', 'Microsec...(older result, shortened)
Step 3, list_recent_deployments: {"window_minutes":15,"moves":[{"at":"2026-09-30T16:41:10.092+00:00","service":"orders","kind":"deploy","version":"26 -> 27","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":true,"settings_changed":[]}}]}

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

rollback_alias service=orders
Verify code changes in git commit 5085754 for orders service regarding metric unit 'Counts' before redeploying.

Hypotheses

likely: bad_deploy caused the orders function to fail due to invalid metric unit 'Counts'
ruled_out: config_regression
ruled_out: timeout_regression
ruled_out: slow_dependency
ruled_out: throttling
ruled_out: retry_storm

What the investigation cost

4 model calls, 14,292 input and 587 output tokens (14,879 total)
6 steps, 12 s of wall clock
222,096 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

orders / bad deploy, confidence 95. right

A new deployment of the orders service (version 27) introduced unhandled exceptions or logic errors, causing 100% of Lambda invocations to fail immediately after the deploy. No evidence of slow dependencies, timeouts, or configuration issues.

Model

ministral-14b-latest

Ended because

finished

Model calls, tokens

6, 26.6K

Logs scanned

197 KB

Hypotheses at the end

likely bad_deploy
ruled out slow_dependency
ruled out config_regression
ruled out timeout_regression

Journal

9 steps

Step 1+92 sget_alarm

name=nightshift-orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T16:42:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T16:42:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+92 sget_metricscited as evidence

namespace=AWS/Lambda, metric=Errors, statistic=Sum, dimensions=FunctionName=nightshift-orders, minutes=15, period=60

Result (285 characters)

{
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "16:38:00Z",
        0
      ],
      [
        "16:39:00Z",
        0
      ],
      [
        "16:40:00Z",
        0
      ],
      [
        "16:41:00Z",
        46
      ]
    ],
    "summary": {
      "min": 0,
      "max": 46,
      "count": 4
    }
  }
}

Step 3+93 slist_recent_deploymentscited as evidence

service=orders, minutes=15

Result (302 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 15,
    "moves": [
      {
        "at": "2026-09-30T16:41:10.092+00:00",
        "service": "orders",
        "kind": "deploy",
        "version": "26 -> 27",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": true,
          "settings_changed": []
        }
      }
    ]
  }
}

Step 4+100 squery_logsfound nothing
service=orders, query=fields @timestamp, level, message, errorType, correlation_id | filter level = "ERROR" and @timestamp >= "2026-09-30T16:40:00Z" and @timestamp <= "2026-09-30T16:42:00Z" | sort @timestamp desc, minutes=2, limit=20
The **nightshift-orders** Lambda function began logging **46 errors in one minute** at **16:41**, exactly when a new deployment (version 27) was rolled out. This strongly suggests the errors are caused by the new code. ### Next steps: - Check the **error logs** for the **orders** service around 16:41 to confirm the nature of the failures (e.g., exceptions, timeouts, or dependency issues). - Verify if the errors are consistent with a **bad_deploy** (e.g., new code path failures) or a **slow_dependency** (e.g., downstream service timeouts). I will now: 4. Query the **orders** service logs for errors during the affected minute. 5. Check the **X-Ray traces** for the **orders** service to identify faults or throttles. This will confirm whether the errors are due to the new code or an external dependency.
Result (145 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 108284,
    "scan_budget_left_bytes": 20863236
  }
}
```

Step 5+101 sget_tracescited as evidence

service=orders, minutes=2

Result (603 characters)

{
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 116,
    "with_error": 94,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.306,
      "p99": 2.68,
      "max": 3.746
    },
    "slowest": [
      {
        "id": "1-6abd3ba6-7526f8dc3324951437f4ec7b",
        "seconds": 3.746
      },
      {
        "id": "1-6abd3ba7-30fc9bb4683e5298625a2a30",
        "seconds": 2.969
      },
      {
        "id": "1-6abd3ba5-5354b62518cbdbe3125b7951",
        "seconds": 2.68
      },
      {
        "id": "1-6abd3bf8-73003b6f110a6e921b16b9e0",
        "seconds": 0.969
      },
      {
        "id": "1-6abd3b9c-4b5e93525d0af00661480e2d",
        "seconds": 0.963
      }
    ]
  }
}

Step 6+109 sget_tracestool returned an error
service=orders, minutes=1, start_time=2026-09-30T16:41:00Z
### Analysis: 1. **No errors in logs** (Step 4): Despite 46 Lambda errors, the **orders** service logs show **no ERROR-level entries** during the affected window. This suggests the errors may be: - **Internal Lambda failures** (e.g., cold starts, initialization errors, or unlogged exceptions). - **Downstream failures** (e.g., timeouts or errors in dependencies like `payments` or `fulfillment`) that are not surfacing in the logs. 2. **X-Ray traces show 94 errors out of 116 traces** (Step 5): - The high error rate (81%) aligns with the Lambda error spike. - **No faults or throttles** are reported, but the **p99 duration (2.68s)** and **max duration (3.746s)** exceed the Lambda timeout (likely 3s or 6s). This suggests **timeouts or slow dependencies** are causing failures. - The errors are not logged in CloudWatch, so they may be **internal Lambda exceptions** or **unhandled downstream timeouts**. 3. **Deployment at 16:41** (Step 3): - The new code (version 27) was dep
Result (164 characters)
```
{
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "error": "invalid arguments",
    "problems": [
      "unknown argument 'start_time'; allowed: ['minutes', 'service']"
    ]
  }
}
```

Step 7+111 sget_tracescited as evidence

service=orders, minutes=1

Result (604 characters)

{
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 60,
    "with_error": 60,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.302,
      "p99": 0.392,
      "max": 0.969
    },
    "slowest": [
      {
        "id": "1-6abd3bf8-73003b6f110a6e921b16b9e0",
        "seconds": 0.969
      },
      {
        "id": "1-6abd3be3-424995f0551c761a793692d3",
        "seconds": 0.392
      },
      {
        "id": "1-6abd3c01-42fea17f077e36c018071636",
        "seconds": 0.361
      },
      {
        "id": "1-6abd3be1-472d30e05716a24a66fee44f",
        "seconds": 0.339
      },
      {
        "id": "1-6abd3bfa-01a6051051b355907c4d348c",
        "seconds": 0.337
      }
    ]
  }
}

Step 8+122 squery_logsfound nothing
service=orders, query=fields @timestamp, correlation_id, message, errorType, stackTrace | filter @timestamp >= "2026-09-30T16:41:00Z" and @timestamp <= "2026-09-30T16:42:00Z" | limit 10, minutes=1
### Updated Analysis: - **Before the deploy (Step 2, 16:38–16:40)**: No Lambda errors. - **After the deploy (Step 5, 16:41–16:42)**: - **94/116 traces failed** (81% error rate). - **P99 duration = 2.68s**, max = 3.746s (likely exceeding Lambda timeout). - **No faults or throttles**, but errors are unlogged in CloudWatch. - **Immediately after deploy (Step 7, 16:41–16:42)**: - **60/60 traces failed** (100% error rate). - **P99 duration = 0.392s**, max = 0.969s (no timeout issues here, but all traces failed). - This suggests the errors are **not timeouts** but **unhandled exceptions or dependency failures** introduced by the new code. ### Next Steps: 8. Check the **slowest traces** (Step 5) for **subsegments** (e.g., calls to `payments` or `fulfillment`) to confirm if the failures are due to **slow dependencies** or **internal logic errors**. 9. Review the **deployment diff** (commit `5085754`) for changes that could cause **unhandled exceptions** or **dependency issues**.
Result (144 characters)
```
{
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 93503,
    "scan_budget_left_bytes": 20769733
  }
}
```
Step 9+130 sfinish_investigation
root_cause_component=orders, fault_category=bad_deploy, confidence=95, summary=A new deployment of the orders service (version 27) introduced unhandled exceptions or logic errors, causing 100% of Lambda invocations to fail immediately after the deploy. No evidence of slow dependencies, timeouts, or configuration issues., evidence_steps=2,3,5,7, hypotheses=likely: bad_deploy ruled_out: slow_dependency ruled_out: config_regression ruled_out: timeout_regression, actions=rollback_alias service=orders, proposed_actions=Review commit 5085754 for the root cause of the failures.
### Conclusion of Investigation: - **No logs** (Step 8) confirm the errors are **internal Lambda exceptions** (e.g., unhandled code paths) or **unlogged dependency failures**. - **100% of traces failed** immediately after the deploy (Step 7), with **no timeout or throttling issues**, pointing to **new code introducing failures**. - **No evidence of slow dependencies** (Step 5/7 show no external faults, just internal errors). - **No recent config changes** (Step 3 shows only a code deploy, no setting changes). ### Hypotheses: - **likely: bad_deploy**: New code (version 27) introduced unhandled exceptions or logic errors. - **ruled_out: slow_dependency**: No external faults or timeouts in traces. - **ruled_out: config_regression**: No setting changes. - **ruled_out: timeout_regression**: No timeout-related evidence. ### Evidence: - Step 2 (Lambda errors spike at deploy time). - Step 3 (Deploy introduced new code). - Step 5/7 (100% trace failures, no external faults). ### Proposed Acti
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 1b6de169c9d7, mistral ministral-14b-latest, started 2026-09-30T16:42:37+00:00.

Answer


Root cause component	`orders`
Fault category	`bad_deploy`
Confidence	95
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T16:42:39+00:00 step 1: get_alarm name=nightshift-orders-errors
2026-09-30T16:42:39+00:00 step 2: get_metrics namespace=AWS/Lambda, metric=Errors, statistic=Sum, dimensions=FunctionName=nightshift-ord...
2026-09-30T16:42:40+00:00 step 3: list_recent_deployments service=orders, minutes=15
2026-09-30T16:42:47+00:00 step 4: query_logs service=orders, query=fields @timestamp, level, message, errorType, correlation_id | filte... (found nothing)
2026-09-30T16:42:48+00:00 step 5: get_traces service=orders, minutes=2
2026-09-30T16:42:56+00:00 step 6: get_traces service=orders, minutes=1, start_time=2026-09-30T16:41:00Z (failed)
2026-09-30T16:42:58+00:00 step 7: get_traces service=orders, minutes=1
2026-09-30T16:43:09+00:00 step 8: query_logs service=orders, query=fields @timestamp, correlation_id, message, errorType, stackTrace | ... (found nothing)
2026-09-30T16:43:17+00:00 step 9: finish_investigation root_cause_component=orders, fault_category=bad_deploy, confidence=95, summary=A new deplo...

Root cause evidence

Step 2, get_metrics: {"metric":"AWS/Lambda Errors {'FunctionName': 'nightshift-orders'} Sum per 60s","window_minutes":15,"points":[["16:38:00Z",0.0],["16:39:00Z",0.0],["16:40:00Z",0.0],["16:41:00Z",46.0]],"summary":{"min":0.0,"max":46.0,"count":4}}
Step 3, list_recent_deployments: {"window_minutes":15,"moves":[{"at":"2026-09-30T16:41:10.092+00:00","service":"orders","kind":"deploy","version":"26 -> 27","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":true,"settings_changed":[]}}]}
Step 5, get_traces: {"service":"nightshift-orders","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":116,"with_error":94,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.306,"p99":2.68,"max":3.746},"slowest":...(older result, shortened)
Step 7, get_traces: {"service":"nightshift-orders","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":60,"with_error":60,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.302,"p99":0.392,"max":0.969},"slowest":...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

rollback_alias service=orders
Review commit 5085754 for the root cause of the failures.

Hypotheses

likely: bad_deploy
ruled_out: slow_dependency
ruled_out: config_regression
ruled_out: timeout_regression

What the investigation cost

6 model calls, 24,596 input and 2,007 output tokens (26,603 total)
9 steps, 42 s of wall clock
201,787 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

orders / insufficient evidence, confidence 50. hedged

An alarm fired indicating errors in the nightshift-orders Lambda function (44 errors recorded), but insufficient information is available from the alarm alone to determine the specific fault category or root cause.

Model

gemini-3.5-flash-lite

Ended because

finished

Model calls, tokens

1, 1.4K

Logs scanned

0 KB

Hypotheses at the end

ruled out bad_deploy
ruled out config_regression
ruled out timeout_regression
ruled out slow_dependency
ruled out poison_message
ruled out iam_regression
ruled out hot_row_contention
ruled out missing_index
ruled out throttling
ruled out retry_storm
ruled out no_fault
possible insufficient_evidence

Journal

2 steps

Step 1+90 sget_alarmfrom the pagecited as evidence

name=orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T16:42:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T16:42:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+95 sfinish_investigation
actions=, fault_category=insufficient_evidence, proposed_actions=Investigate CloudWatch logs and metrics for the nightshift-orders Lambda function to determine the root cause of the errors., confidence=50, root_cause_component=orders, summary=An alarm fired indicating errors in the nightshift-orders Lambda function (44 errors recorded), but insufficient information is available from the alarm alone to determine the specific fault category or root cause., evidence_steps=1, hypotheses=ruled_out: bad_deploy ruled_out: config_regression ruled_out: timeout_regression ruled_out: slow_dependency ruled_out: poison_message ruled_out: iam_regression ruled_out: hot_row_contention ruled_out: missing_index ruled_out: throttling ruled_out: retry_storm ruled_out: no_fault possible: insufficient_evidence
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation fc976f066952, gemini gemini-3.5-flash-lite, started 2026-09-30T16:42:37+00:00.

Answer


Root cause component	`orders`
Fault category	`insufficient_evidence`
Confidence	50
Ended because	finished

Summary

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T16:42:37+00:00 step 1: get_alarm name=orders-errors
2026-09-30T16:42:42+00:00 step 2: finish_investigation actions=, fault_category=insufficient_evidence, proposed_actions=Investigate CloudWatch lo...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).","since":"2026-09-30T16:42:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

Investigate CloudWatch logs and metrics for the nightshift-orders Lambda function to determine the root cause of the errors.

Hypotheses

ruled_out: bad_deploy
ruled_out: config_regression
ruled_out: timeout_regression
ruled_out: slow_dependency
ruled_out: poison_message
ruled_out: iam_regression
ruled_out: hot_row_contention
ruled_out: missing_index
ruled_out: throttling
ruled_out: retry_storm
ruled_out: no_fault
possible: insufficient_evidence

What the investigation cost

1 model calls, 1,187 input and 225 output tokens (1,412 total)
2 steps, 5 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

none / insufficient evidence, confidence 95. hedged

The alarm indicates a sudden spike in Lambda errors for the `nightshift-orders` function, but no additional context or tool results are available to determine the root cause. The error rate jumped from 0 to 44 errors in a single minute, which is abnormal but lacks specific details about the nature of the errors or their impact on downstream services.

Model

ministral-14b-latest

Ended because

finished

Model calls, tokens

1, 1.6K

Logs scanned

0 KB

Hypotheses at the end

likely The spike in errors could be due to a transient issue like a throttling event, a sudden increase in malformed input, or a dependency failure.
possible A recent deployment or configuration change might have introduced a bug or regression.
possible The Lambda function could be experiencing throttling due to sudden traffic or concurrency limits.
possible A downstream service (e.g., database, API, or queue) might be slow or unavailable, causing the Lambda to fail.
ruled out No evidence yet to confirm a poison message, IAM regression, or hot row contention.
ruled out No evidence of a retry storm or missing index.

Journal

2 steps

Step 1+90 sget_alarmfrom the pagecited as evidence

name=orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T16:42:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T16:42:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+96 sfinish_investigation
root_cause_component=none, fault_category=insufficient_evidence, confidence=95, summary=The alarm indicates a sudden spike in Lambda errors for the `nightshift-orders` function, but no additional context or tool results are available to determine the root cause. The error rate jumped from 0 to 44 errors in a single minute, which is abnormal but lacks specific details about the nature of the errors or their impact on downstream services., evidence_steps=1, hypotheses=likely: The spike in errors could be due to a transient issue like a throttling event, a sudden increase in malformed input, or a dependency failure. possible: A recent deployment or configuration change might have introduced a bug or regression. possible: The Lambda function could be experiencing throttling due to sudden traffic or concurrency limits. possible: A downstream service (e.g., database, API, or queue) might be slow or unavailable, causing the Lambda to fail. ruled_out: No evidence yet to confirm a poison message, IAM regression, or hot row contention. ruled_out: No evidence of a retry storm or missing index., actions=, proposed_actions=1. Check CloudWatch Logs for the `nightshift-orders` Lambda to identify error patterns or stack traces. 2. Review recent deployments or configuration changes for the Lambda function or its dependencies. 3. Verify if there are any related alarms or metrics for downstream services (e.g., DynamoDB, SQS, or other APIs). 4. Check Lambda concurrency metrics to rule out throttling. 5. Investigate if the errors are related to specific input data or triggers.
Result (30 characters)
```
{
  "ok": "investigation closed"
}
```

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 9f184dfa1433, mistral ministral-14b-latest, started 2026-09-30T16:42:37+00:00.

Answer


Root cause component	`none`
Fault category	`insufficient_evidence`
Confidence	95
Ended because	finished

Summary

The alarm indicates a sudden spike in Lambda errors for the nightshift-orders function, but no additional context or tool results are available to determine the root cause. The error rate jumped from 0 to 44 errors in a single minute, which is abnormal but lacks specific details about the nature of the errors or their impact on downstream services.

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T16:42:37+00:00 step 1: get_alarm name=orders-errors
2026-09-30T16:42:43+00:00 step 2: finish_investigation root_cause_component=none, fault_category=insufficient_evidence, confidence=95, summary=Th...

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).","since":"2026-09-30T16:42:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

1. Check CloudWatch Logs for the nightshift-orders Lambda to identify error patterns or stack traces.
1. Review recent deployments or configuration changes for the Lambda function or its dependencies.
1. Verify if there are any related alarms or metrics for downstream services (e.g., DynamoDB, SQS, or other APIs).
1. Check Lambda concurrency metrics to rule out throttling.
1. Investigate if the errors are related to specific input data or triggers.

Hypotheses

likely: The spike in errors could be due to a transient issue like a throttling event, a sudden increase in malformed input, or a dependency failure.
possible: A recent deployment or configuration change might have introduced a bug or regression.
possible: The Lambda function could be experiencing throttling due to sudden traffic or concurrency limits.
possible: A downstream service (e.g., database, API, or queue) might be slow or unavailable, causing the Lambda to fail.
ruled_out: No evidence yet to confirm a poison message, IAM regression, or hot row contention.
ruled_out: No evidence of a retry storm or missing index.

What the investigation cost

1 model calls, 1,192 input and 375 output tokens (1,567 total)
2 steps, 6 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments

orders / bad deploy, confidence 70. right

orders was deployed at 2026-09-30T16:41:10.092+00:00 (code changed) before orders-errors fired. Roll it back.

Model

scripted

Ended because

finished

Model calls, tokens

0, 0

Logs scanned

0 KB

Hypotheses at the end

likely orders was deployed at 2026-09-30T16:41:10.092+00:00 (code changed) before orders-errors fired. Roll it back.

Journal

2 steps

Step 1+90 sget_alarmcited as evidence

name=orders-errors

Result (560 characters)

{
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-09-30T16:42:31Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-09-30T16:42:31Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}

Step 2+91 slist_recent_deploymentscited as evidence

minutes=30, service=orders

Result (302 characters)

{
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": [
      {
        "at": "2026-09-30T16:41:10.092+00:00",
        "service": "orders",
        "kind": "deploy",
        "version": "26 -> 27",
        "git_sha": "5085754",
        "actor": "youssef",
        "reason": null,
        "changed": {
          "code_changed": true,
          "settings_changed": []
        }
      }
    ]
  }
}

Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 5e002ac02b65, runbook scripted, started 2026-09-30T16:42:37+00:00.

Answer


Root cause component	`orders`
Fault category	`bad_deploy`
Confidence	70
Ended because	finished

Summary

orders was deployed at 2026-09-30T16:41:10.092+00:00 (code changed) before orders-errors fired. Roll it back.

Impact

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

2026-09-30T16:42:37+00:00 step 1: get_alarm name=orders-errors
2026-09-30T16:42:38+00:00 step 2: list_recent_deployments minutes=30, service=orders

Root cause evidence

Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [44.0 (30/09/26 16:41:00)] was greater than or equal to the threshold (1.0).","since":"2026-09-30T16:42:31Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)
Step 2, list_recent_deployments: {"window_minutes":30,"moves":[{"at":"2026-09-30T16:41:10.092+00:00","service":"orders","kind":"deploy","version":"26 -> 27","git_sha":"5085754","actor":"youssef","reason":null,"changed":{"code_changed":true,"settings_changed":[]}}]}

Proposed fix (for a human to decide; the agent changed nothing)

Allowlisted actions, each waiting for owner approval:

rollback_alias service=orders

Hypotheses

likely: orders was deployed at 2026-09-30T16:41:10.092+00:00 (code changed) before orders-errors fired. Roll it back.

What the investigation cost

0 model calls, 0 input and 0 output tokens (0 total)
2 steps, 0 s of wall clock
0 bytes of logs scanned
0 replies rejected by the provider for bad tool arguments