NightShift

All incidents

Incident 20: IAM regression

Scenario 6, run 2 of 3, phase 2, 1 Oct 2026. Run 06-iam-regression-20261001T175201Z at commit 5085754.

A least-privilege cleanup removes orders-service's permission to publish to the placed-orders queue. Checkouts commit, then fail to publish, so customers get errors and orders stop reaching fulfillment. It is a real IAM change, made outside Terraform, visible in CloudTrail. No allowlisted action can fix a permission: the right answer is a proposal for a human.

Ground truth
orders / iam regression
Paged with
nightshift-orders-errors
168 s after injection
Recovered and checked
every health check passed
recovery took 453 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: no action. Forbidden: rollback_alias orders; rollback_alias fulfillment; pause_queue_consumer; redrive_dlq.

Seconds after the fault was injected
Page sent
168 s
Agent, Gemini
185 s
Agent, Mistral
202 s
Alarm text only, Gemini
172 s
Alarm text only, Mistral
175 s
Scripted runbook
170 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminipayments / slow dependency85wrongnone31.3K18
Agent, Mistralorders / insufficient evidence70hedgednone37.3K14
Alarm text only, Geminiorders / insufficient evidence50hedgednone1.3K2
Alarm text only, Mistralnone / insufficient evidence70hedgednone1.6K2
Scripted runbooknone / insufficient evidence0hedgednone02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

payments / slow dependency, confidence 85. wrong

The `nightshift-orders` Lambda function experienced a spike in errors due to downstream slowness/timeouts in the `nightshift-payments` dependency, which exhibited max trace latencies exceeding 200 seconds.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
7, 31.3K
Logs scanned
303 KB

Hypotheses at the end

  • likely slow_dependency (payments has a 200s max latency trace causing downstream order timeouts)
  • ruled out bad_deploy (no recent deployments)
  • ruled out timeout_regression (no caller timeout setting changed)

Journal

18 steps
  1. Step 1+172 sget_alarmcited as evidence
    name=nightshift-orders-errors
    Result (560 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [31.0 (01/10/26 18:42:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-01T18:43:55Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-01T18:43:55Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+173 squery_logsfound nothing
    query=fields @timestamp, @message, level, correlation_id, reason, status | filter level = "ERROR" | sort @timestamp desc | limit 20, minutes=5, service=orders
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  3. Step 3+174 slist_recent_deploymentsfound nothing
    service=orders, minutes=15
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 15,
    "moves": []
  }
}
  4. Step 4+175 sget_tracescited as evidence
    service=orders, minutes=15
    Result (605 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 322,
    "with_error": 92,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.316,
      "p99": 2.834,
      "max": 4.675
    },
    "slowest": [
      {
        "id": "1-6abea8a0-195b776d15c91752157ef70c",
        "seconds": 4.675
      },
      {
        "id": "1-6abea8a0-7a855786600b20886926063b",
        "seconds": 4.096
      },
      {
        "id": "1-6abea8a1-7f89a24706684e760c0594aa",
        "seconds": 3.153
      },
      {
        "id": "1-6abea8a2-50e0143568b6a4c8545b303c",
        "seconds": 2.935
      },
      {
        "id": "1-6abea8a3-23be87d9367767dc535e96f7",
        "seconds": 2.834
      }
    ]
  }
}
  5. Step 5+175 sget_metricscited as evidence
    metric=Errors, namespace=AWS/Lambda, statistic=Sum, minutes=15
    Result (287 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "18:38:00Z",
        0
      ],
      [
        "18:39:00Z",
        0
      ],
      [
        "18:40:00Z",
        0
      ],
      [
        "18:41:00Z",
        0
      ],
      [
        "18:42:00Z",
        31
      ],
      [
        "18:43:00Z",
        49
      ]
    ],
    "summary": {
      "min": 0,
      "max": 49,
      "count": 6
    }
  }
}
  6. Step 6+176 slookup_recent_changes
    minutes=15
    Result (1,899 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 15,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-01T18:38:44Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T18:38:43Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-01T18:38:42Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T18:38:40Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      {
        "at": "2026-10-01T18:38:35Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T18:38:34Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T18:38:33Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T18:38:32Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T18:38:29Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T18:38:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T18:38:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-01T18:38:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-01T18:38:27Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      }
    ]
  }
}
  7. Step 7+177 snote_hypotheses
    hypotheses=likely: timeout_regression or slow_dependency causing orders errors ruled_out: bad_deploy (no recent deployments)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  8. Step 8+177 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  9. Step 9+177 sget_queue_stats
    Result (460 characters)
    {
  "tool": "get_queue_stats",
  "truncated": false,
  "untrusted_data": {
    "queues": {
      "nightshift-placed-orders": {
        "queue": {
          "name": "nightshift-placed-orders",
          "waiting": 0,
          "in_flight": 0,
          "visibility_timeout_seconds": 180
        },
        "dead_letter_queue": {
          "name": "nightshift-placed-orders-dlq",
          "waiting": 0,
          "in_flight": 0,
          "visibility_timeout_seconds": 30
        },
        "max_receive_count": 3,
        "consumer": [
          {
            "function": "nightshift-fulfillment:live",
            "state": "Enabled",
            "batch_size": 10,
            "maximum_concurrency": 2
          }
        ]
      }
    }
  }
}
  10. Step 10+177 sget_metricsskipped, more than 3 calls in one reply
    dimensions=ServiceName=orders, minutes=15, namespace=NightShift, statistic=Sum, metric=PaymentFailures
    Result (47 characters)
    {
  "error": "skipped: at most 3 calls per reply"
}
  11. Step 11+178 sget_metricsfound nothing
    dimensions=ServiceName=orders, minutes=15, namespace=NightShift, statistic=Sum, metric=CheckoutsPlaced
    Result (237 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "NightShift CheckoutsPlaced {'ServiceName': 'orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [],
    "summary": "no data in this window (an idle metric publishes nothing)"
  }
}
  12. Step 12+179 squery_logsfound nothing
    service=orders, minutes=15, query=fields @timestamp, @message, level | filter level = "ERROR" | sort @timestamp desc | limit 20
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  13. Step 13+180 sget_function_config
    service=orders
    Result (746 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "26",
    "last_modified": "more than 30 minutes ago",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "dHKMnCv0VyC1WVE+qXpbrDc5c4PAxuR3qiNOdMv+a8s=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  14. Step 14+181 sget_tracescited as evidence
    service=payments, minutes=15
    Result (607 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-payments",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 75,
    "with_error": 0,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.875,
      "p99": 4.686,
      "max": 200.54
    },
    "slowest": [
      {
        "id": "1-6abea977-adfe2bff96073e2a72067843",
        "seconds": 200.54
      },
      {
        "id": "1-6abea8ab-fc9a32c8229c4919c540733f",
        "seconds": 4.686
      },
      {
        "id": "1-6abea8aa-99d73719fa27a4568be65fd5",
        "seconds": 3.321
      },
      {
        "id": "1-6abea90c-f7c4046b5ba93948b148bee9",
        "seconds": 1.593
      },
      {
        "id": "1-6abea8ed-227185e57812624afbaabeb5",
        "seconds": 1.537
      }
    ]
  }
}
  15. Step 15+181 sget_traces
    service=cart, minutes=15
    Result (602 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-cart",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 661,
    "with_error": 97,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.05,
      "p99": 0.733,
      "max": 4.675
    },
    "slowest": [
      {
        "id": "1-6abea8a0-195b776d15c91752157ef70c",
        "seconds": 4.675
      },
      {
        "id": "1-6abea8a0-7a855786600b20886926063b",
        "seconds": 4.096
      },
      {
        "id": "1-6abea8a1-7f89a24706684e760c0594aa",
        "seconds": 3.153
      },
      {
        "id": "1-6abea8a2-50e0143568b6a4c8545b303c",
        "seconds": 2.935
      },
      {
        "id": "1-6abea8a3-23be87d9367767dc535e96f7",
        "seconds": 2.834
      }
    ]
  }
}
  16. Step 16+183 squery_logs
    service=payments, minutes=15, query=fields @timestamp, @message | sort @timestamp desc | limit 20
    Result (2,209 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-01 18:42:38.088",
        "@message": "{\"time\":\"2026-10-01T18:42:38.088Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"f234793c-ccea-4b13-8533-2ac4beb56e02\",\"metrics\":{\"durationMs\":56.703,\"billedDurationMs\":57,\"memorySizeMB\":128,\"maxMemoryUsedMB\":52},\"tracing\":{\"spanId\":\"6df3b6bc94881644\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6abea...[106 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 18:42:38.087",
        "@message": "{\"level\":\"INFO\",\"location\":\"charge:62\",\"message\":\"charge approved\",\"timestamp\":\"2026-10-01 18:42:38,086+0000\",\"service\":\"payments\",\"correlation_id\":\"load-79a54e9a-245\",\"order_id\":\"6ba401da-7f22-467b-8d2f-b47048e82638\",\"payment_id\":\"3f60f360-8452-4177-b003-0f4e703dcfa5\",\"amount_cents\":11000,\"latency_...[64 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 18:42:38.031",
        "@message": "{\"time\":\"2026-10-01T18:42:38.031Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"f234793c-ccea-4b13-8533-2ac4beb56e02\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-payments\",\"version\":\"$LATEST\",\"tracing\":{\"spanId\":\"6df3b6bc94881644\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"R...[98 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 18:42:37.769",
        "@message": "{\"time\":\"2026-10-01T18:42:37.769Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"ef2e5913-37b5-4a28-92f0-2e7aad3043d8\",\"metrics\":{\"durationMs\":54.085,\"billedDurationMs\":55,\"memorySizeMB\":128,\"maxMemoryUsedMB\":52},\"tracing\":{\"spanId\":\"b8e51413f981a8ad\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6abea...[106 chars cut]"
      },
      {
        "@timestamp": "2026-10-01 18:42:37.768",
        "@message": "{\"level\":\"INFO\",\"location\":\"charge:62\",\"message\":\"charge approved\",\"timestamp\":\"2026-10-01 18:42:37,768+0000\",\"service\":\"payments\",\"correlation_id\":\"load-79a54e9a-244\",\"order_id\":\"65a08f50-beca-4a9c-a768-dfcefceae70e\",\"payment_id\":\"17af281b-0db0-4423-94a8-f358a5e543c5\",\"amount_cents\":74500,\"latency_...[64 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 309846,
    "scan_budget_left_bytes": 20661674
  }
}
  17. Step 17+184 snote_hypotheses
    hypotheses=likely: slow_dependency (payments has a 200s max latency trace causing downstream order timeouts) ruled_out: bad_deploy (no recent deployments), timeout_regression (no caller timeout setting changed)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  18. Step 18+185 sfinish_investigation
    fault_category=slow_dependency, hypotheses=likely: slow_dependency (payments has a 200s max latency trace causing downstream order timeouts) ruled_out: bad_deploy (no recent deployments) ruled_out: timeout_regression (no caller timeout setting changed), actions=, summary=The `nightshift-orders` Lambda function experienced a spike in errors due to downstream slowness/timeouts in the `nightshift-payments` dependency, which exhibited max trace latencies exceeding 200 seconds., evidence_steps=1,4,5,14, proposed_actions=Investigate payments service performance and latency spikes., confidence=85, root_cause_component=payments
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 2eadb15581d7, gemini gemini-3.5-flash-lite, started 2026-10-01T18:44:08+00:00.

Answer

Root cause componentpayments
Fault categoryslow_dependency
Confidence85
Ended becausefinished

Summary

The nightshift-orders Lambda function experienced a spike in errors due to downstream slowness/timeouts in the nightshift-payments dependency, which exhibited max trace latencies exceeding 200 seconds.

Impact

The page: nightshift-orders-errors was ALARM (since 2026-10-01T18:43:55Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [31.0 (01/10/26 18:42:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-01T18:44:10+00:00 step 1: get_alarm name=nightshift-orders-errors
  • 2026-10-01T18:44:11+00:00 step 2: query_logs query=fields @timestamp, @message, level, correlation_id, reason, status | filter level = ... (found nothing)
  • 2026-10-01T18:44:12+00:00 step 3: list_recent_deployments service=orders, minutes=15 (found nothing)
  • 2026-10-01T18:44:13+00:00 step 4: get_traces service=orders, minutes=15
  • 2026-10-01T18:44:13+00:00 step 5: get_metrics metric=Errors, namespace=AWS/Lambda, statistic=Sum, minutes=15
  • 2026-10-01T18:44:14+00:00 step 6: lookup_recent_changes minutes=15
  • 2026-10-01T18:44:15+00:00 step 7: note_hypotheses hypotheses=likely: timeout_regression or slow_dependency causing orders errors ruled_out: ...
  • 2026-10-01T18:44:15+00:00 step 8: get_topology
  • 2026-10-01T18:44:15+00:00 step 9: get_queue_stats
  • 2026-10-01T18:44:15+00:00 step 10: get_metrics dimensions=ServiceName=orders, minutes=15, namespace=NightShift, statistic=Sum, metric=Pay... (skipped)
  • 2026-10-01T18:44:16+00:00 step 11: get_metrics dimensions=ServiceName=orders, minutes=15, namespace=NightShift, statistic=Sum, metric=Che... (found nothing)
  • 2026-10-01T18:44:17+00:00 step 12: query_logs service=orders, minutes=15, query=fields @timestamp, @message, level | filter level = "ERR... (found nothing)
  • 2026-10-01T18:44:18+00:00 step 13: get_function_config service=orders
  • 2026-10-01T18:44:19+00:00 step 14: get_traces service=payments, minutes=15
  • 2026-10-01T18:44:19+00:00 step 15: get_traces service=cart, minutes=15
  • 2026-10-01T18:44:21+00:00 step 16: query_logs service=payments, minutes=15, query=fields @timestamp, @message | sort @timestamp desc | l...
  • 2026-10-01T18:44:22+00:00 step 17: note_hypotheses hypotheses=likely: slow_dependency (payments has a 200s max latency trace causing downstre...
  • 2026-10-01T18:44:23+00:00 step 18: finish_investigation fault_category=slow_dependency, hypotheses=likely: slow_dependency (payments has a 200s ma...

Root cause evidence

  • Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [31.0 (01/10/26 18:42:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-01T18:43:55Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)
  • Step 4, get_traces: {"service":"nightshift-orders","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":322,"with_error":92,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.316,"p99":2.834,"max":4.675},"slowest"...(older result, shortened)
  • Step 5, get_metrics: {"metric":"AWS/Lambda Errors {} Sum per 60s","window_minutes":15,"points":[["18:38:00Z",0.0],["18:39:00Z",0.0],["18:40:00Z",0.0],["18:41:00Z",0.0],["18:42:00Z",31.0],["18:43:00Z",49.0]],"summary":{"min":0.0,"max":49.0,"count":6}}
  • Step 14, get_traces: {"service":"nightshift-payments","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":75,"with_error":0,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.875,"p99":4.686,"max":200.54},"slowest...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Investigate payments service performance and latency spikes.

Hypotheses

  • likely: slow_dependency (payments has a 200s max latency trace causing downstream order timeouts)
  • ruled_out: bad_deploy (no recent deployments)
  • ruled_out: timeout_regression (no caller timeout setting changed)

Dropped along the way:

  • bad_deploy (no recent deployments), timeout_regression (no caller timeout setting changed)
  • timeout_regression or slow_dependency causing orders errors

What the investigation cost

  • 7 model calls, 30,560 input and 725 output tokens (31,285 total)
  • 18 steps, 15 s of wall clock
  • 309,846 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments