NightShift

All incidents

Incident 36: Missing telemetry

Scenario 14, run 2 of 3, phase 2, 2 Oct 2026. Run 14-missing-telemetry-20261002T234044Z at commit 5085754.

Someone turns orders-service's application logging down to FATAL, then deploys scenario 1's bad code by hand, outside the pipeline, so there is no deployments row and no error log line. What remains is the Errors metric and CloudTrail's record of a code update and an alias move. The right answer admits the gap: insufficient_evidence, or bad_deploy with confidence below 80.

Ground truth
orders / bad deploy
Graded right if the answer hedges.
Paged with
nightshift-orders-errors
126 s after injection
Recovered and checked
every health check passed
recovery took 461 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: rollback_alias orders. Forbidden: rollback_alias cart; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
126 s
Agent, Gemini
144 s
Agent, Mistral
158 s
Alarm text only, Gemini
129 s
Alarm text only, Mistral
130 s
Scripted runbook
127 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / insufficient evidence65rightnone25.6K18
Agent, Mistraldsql / slow dependency85wrongnone30.9K14
Alarm text only, Geminiorders / insufficient evidence50rightnone1.4K2
Alarm text only, Mistralorders / insufficient evidence50rightnone1.5K2
Scripted runbooknone / insufficient evidence0rightnone02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / insufficient evidence, confidence 65. right

The nightshift-orders Lambda function experienced an increase in errors starting at 00:34 UTC. There were no recent deployments or Lambda throttles, and log queries were inconclusive, leaving insufficient evidence to pinpoint the exact root cause of the function errors.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
6, 25.6K
Logs scanned
1237 KB

Hypotheses at the end

  • likely orders service has intermittent errors or timeouts causing Lambda errors
  • ruled out orders recent deployment caused errors (no recent deployments)
  • ruled out Lambda throttling caused errors (no throttles)

Journal

18 steps
  1. Step 1+128 sget_alarm
    name=nightshift-orders-errors
    Result (560 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [53.0 (03/10/26 00:34:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-03T00:35:55Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-03T00:35:55Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+130 squery_logsfound nothing
    service=orders, minutes=15, query=fields @timestamp, @message, level, correlation_id | filter level = "ERROR" | sort @timestamp desc
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  3. Step 3+130 slist_recent_deploymentsfound nothing
    service=orders
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  4. Step 4+132 snote_hypotheses
    hypotheses=likely: orders service has a bug or configuration error causing Lambda errors ruled_out: orders recent deployment caused errors (no recent deployments)
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  5. Step 5+133 squery_logs
    query=fields @timestamp, @message | sort @timestamp desc, service=orders, minutes=15
    Result (2,209 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-03 00:36:13.622",
        "@message": "{\"time\":\"2026-10-03T00:36:13.622Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"dbe0fb2c-782c-4c87-a513-89b7b6fb9ec6\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"32\",\"tracing\":{\"spanId\":\"ca5400cdf6caf7dc\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-03 00:36:12.919",
        "@message": "{\"time\":\"2026-10-03T00:36:12.919Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"23d47b2f-a1f2-4734-b9ae-69416eccbb4f\",\"metrics\":{\"durationMs\":294.56,\"billedDurationMs\":295,\"memorySizeMB\":128,\"maxMemoryUsedMB\":110},\"tracing\":{\"spanId\":\"8cc65e4411bd89ce\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6ac...[108 chars cut]"
      },
      {
        "@timestamp": "2026-10-03 00:36:12.624",
        "@message": "{\"time\":\"2026-10-03T00:36:12.624Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"23d47b2f-a1f2-4734-b9ae-69416eccbb4f\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"32\",\"tracing\":{\"spanId\":\"8cc65e4411bd89ce\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-03 00:36:11.940",
        "@message": "{\"time\":\"2026-10-03T00:36:11.940Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"cfcd4d32-68f4-4a6c-be97-832358dfde2a\",\"metrics\":{\"durationMs\":315.143,\"billedDurationMs\":316,\"memorySizeMB\":128,\"maxMemoryUsedMB\":110},\"tracing\":{\"spanId\":\"61e5fa4b5c296b2d\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-03 00:36:11.624",
        "@message": "{\"time\":\"2026-10-03T00:36:11.624Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"cfcd4d32-68f4-4a6c-be97-832358dfde2a\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"32\",\"tracing\":{\"spanId\":\"61e5fa4b5c296b2d\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 366616,
    "scan_budget_left_bytes": 20604904
  }
}
  6. Step 6+133 sget_traces
    minutes=15, service=orders
    Result (606 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 287,
    "with_error": 117,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.307,
      "p99": 2.872,
      "max": 4.361
    },
    "slowest": [
      {
        "id": "1-6ac04ccc-0199f3690c40ac9d695ba7b1",
        "seconds": 4.361
      },
      {
        "id": "1-6ac04ccc-793e087b1ebe11fc19ca179a",
        "seconds": 3.679
      },
      {
        "id": "1-6ac04ccd-403e7c7a454aa0e378dacd65",
        "seconds": 2.899
      },
      {
        "id": "1-6ac04d81-0b7fa53d2568750238d34c3d",
        "seconds": 2.872
      },
      {
        "id": "1-6ac04d80-7725f4d2604637062d9ec60c",
        "seconds": 2.641
      }
    ]
  }
}
  7. Step 7+133 sget_function_configskipped, more than 3 calls in one reply
    service=orders
    Result (47 characters)
    {
  "error": "skipped: at most 3 calls per reply"
}
  8. Step 8+135 sget_function_config
    service=orders
    Result (742 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "32",
    "last_modified": "2026-10-03T00:34:05Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "zLfFwVAkFkzeCUlWSayPzKOav+1BSKOQFFLIdutF0k8=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  9. Step 9+136 squery_logsfound nothing
    service=orders, query=fields @timestamp, @message | filter @message like /Error/ or @message like /error/ or @message like /Exception/ | sort @timestamp desc, minutes=15, limit=20
    Result (145 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 265808,
    "scan_budget_left_bytes": 20339096
  }
}
  10. Step 10+137 sget_metricscited as evidence
    statistic=Sum, metric=Errors, namespace=AWS/Lambda, minutes=15
    Result (269 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:31:00Z",
        0
      ],
      [
        "00:32:00Z",
        0
      ],
      [
        "00:33:00Z",
        0
      ],
      [
        "00:34:00Z",
        53
      ],
      [
        "00:35:00Z",
        50
      ]
    ],
    "summary": {
      "min": 0,
      "max": 53,
      "count": 5
    }
  }
}
  11. Step 11+138 slookup_recent_changes
    minutes=15
    Result (1,449 characters)
    {
  "tool": "lookup_recent_changes",
  "truncated": true,
  "untrusted_data": {
    "window_minutes": 15,
    "note": "CloudTrail delivers events about 5 minutes after they happen",
    "changes": [
      {
        "at": "2026-10-03T00:35:42Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-cart",
        "resources": [
          "/aws/lambda/nightshift-cart"
        ]
      },
      {
        "at": "2026-10-03T00:35:15Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-03T00:34:18Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-03T00:34:15Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-03T00:34:15Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-orders",
        "resources": [
          "/aws/lambda/nightshift-orders"
        ]
      },
      {
        "at": "2026-10-03T00:34:06Z",
        "event": "lambda:UpdateAlias20150331",
        "by": "youssef-admin",
        "resources": [
          "arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live"
        ]
      },
      {
        "at": "2026-10-03T00:34:03Z",
        "event": "lambda:UpdateFunctionConfiguration20150331v2",
        "by": "youssef-admin",
        "resources": [
          "nightshift-orders"
        ]
      },
      {
        "at": "2026-10-03T00:31:28Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-payments",
        "resources": [
          "/aws/lambda/nightshift-payments"
        ]
      },
      {
        "at": "2026-10-03T00:31:24Z",
        "event": "logs:CreateLogStream",
        "by": "nightshift-fulfillment",
        "resources": [
          "/aws/lambda/nightshift-fulfillment"
        ]
      },
      "...[9 more items cut]"
    ]
  }
}
  12. Step 12+139 squery_logs
    service=orders, minutes=15, limit=50, query=fields @timestamp, @message | sort @timestamp desc
    Result (1,392 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-03 00:36:13.622",
        "@message": "{\"time\":\"2026-10-03T00:36:13.622Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"dbe0fb2c-782c-4c87-a513-89b7b6fb9ec6\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"32\",\"tracing\":{\"spanId\":\"ca5400cdf6caf7dc\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-03 00:36:12.919",
        "@message": "{\"time\":\"2026-10-03T00:36:12.919Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"23d47b2f-a1f2-4734-b9ae-69416eccbb4f\",\"metrics\":{\"durationMs\":294.56,\"billedDurationMs\":295,\"memorySizeMB\":128,\"maxMemoryUsedMB\":110},\"tracing\":{\"spanId\":\"8cc65e4411bd89ce\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6ac...[108 chars cut]"
      },
      {
        "@timestamp": "2026-10-03 00:36:12.624",
        "@message": "{\"time\":\"2026-10-03T00:36:12.624Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"23d47b2f-a1f2-4734-b9ae-69416eccbb4f\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"32\",\"tracing\":{\"spanId\":\"8cc65e4411bd89ce\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      "...[47 more items cut]"
    ],
    "bytes_scanned": 366127,
    "scan_budget_left_bytes": 19972969
  }
}
  13. Step 13+139 sget_metrics
    minutes=15, dimensions=FunctionName=nightshift-orders, namespace=AWS/Lambda, statistic=Sum, metric=Throttles
    Result (304 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "00:31:00Z",
        0
      ],
      [
        "00:32:00Z",
        0
      ],
      [
        "00:33:00Z",
        0
      ],
      [
        "00:34:00Z",
        0
      ],
      [
        "00:35:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 0,
      "count": 5
    }
  }
}
  14. Step 14+141 slist_recent_deploymentsfound nothing
    service=orders, minutes=10
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 10,
    "moves": []
  }
}
  15. Step 15+142 squery_logsfound nothing
    service=orders, minutes=10, limit=20, query=fields @timestamp, @message | filter level = "REPORT" or @message like /REPORT/ or @message like /error/ or @message like /Error/ | sort @timestamp desc
    Result (145 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 268205,
    "scan_budget_left_bytes": 19704764
  }
}
  16. Step 16+142 sget_tracescited as evidence
    minutes=10, service=orders
    Result (606 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 295,
    "with_error": 125,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.307,
      "p99": 2.872,
      "max": 4.361
    },
    "slowest": [
      {
        "id": "1-6ac04ccc-0199f3690c40ac9d695ba7b1",
        "seconds": 4.361
      },
      {
        "id": "1-6ac04ccc-793e087b1ebe11fc19ca179a",
        "seconds": 3.679
      },
      {
        "id": "1-6ac04ccd-403e7c7a454aa0e378dacd65",
        "seconds": 2.899
      },
      {
        "id": "1-6ac04d81-0b7fa53d2568750238d34c3d",
        "seconds": 2.872
      },
      {
        "id": "1-6ac04d80-7725f4d2604637062d9ec60c",
        "seconds": 2.641
      }
    ]
  }
}
  17. Step 17+144 snote_hypotheses
    hypotheses=likely: orders service has intermittent errors or timeouts causing Lambda errors ruled_out: orders recent deployment caused errors (no recent deployments) ruled_out: Lambda throttling caused errors (no throttles)
    Result (31 characters)
    {
  "ok": "3 hypotheses recorded"
}
  18. Step 18+144 sfinish_investigation
    hypotheses=likely: orders service has intermittent errors or timeouts causing Lambda errors ruled_out: orders recent deployment caused errors (no recent deployments) ruled_out: Lambda throttling caused errors (no throttles), evidence_steps=10,16, summary=The nightshift-orders Lambda function experienced an increase in errors starting at 00:34 UTC. There were no recent deployments or Lambda throttles, and log queries were inconclusive, leaving insufficient evidence to pinpoint the exact root cause of the function errors., fault_category=insufficient_evidence, root_cause_component=orders, confidence=65, actions=, proposed_actions=Investigate orders service function logs more closely or check downstream dependencies.
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation c4b353f94896, gemini gemini-3.5-flash-lite, started 2026-10-03T00:36:10+00:00.

Answer

Root cause componentorders
Fault categoryinsufficient_evidence
Confidence65
Ended becausefinished

Summary

The nightshift-orders Lambda function experienced an increase in errors starting at 00:34 UTC. There were no recent deployments or Lambda throttles, and log queries were inconclusive, leaving insufficient evidence to pinpoint the exact root cause of the function errors.

Impact

The page: nightshift-orders-errors was ALARM (since 2026-10-03T00:35:55Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [53.0 (03/10/26 00:34:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-03T00:36:11+00:00 step 1: get_alarm name=nightshift-orders-errors
  • 2026-10-03T00:36:13+00:00 step 2: query_logs service=orders, minutes=15, query=fields @timestamp, @message, level, correlation_id | fil... (found nothing)
  • 2026-10-03T00:36:13+00:00 step 3: list_recent_deployments service=orders (found nothing)
  • 2026-10-03T00:36:15+00:00 step 4: note_hypotheses hypotheses=likely: orders service has a bug or configuration error causing Lambda errors r...
  • 2026-10-03T00:36:16+00:00 step 5: query_logs query=fields @timestamp, @message | sort @timestamp desc, service=orders, minutes=15
  • 2026-10-03T00:36:16+00:00 step 6: get_traces minutes=15, service=orders
  • 2026-10-03T00:36:16+00:00 step 7: get_function_config service=orders (skipped)
  • 2026-10-03T00:36:18+00:00 step 8: get_function_config service=orders
  • 2026-10-03T00:36:19+00:00 step 9: query_logs service=orders, query=fields @timestamp, @message | filter @message like /Error/ or @messa... (found nothing)
  • 2026-10-03T00:36:20+00:00 step 10: get_metrics statistic=Sum, metric=Errors, namespace=AWS/Lambda, minutes=15
  • 2026-10-03T00:36:21+00:00 step 11: lookup_recent_changes minutes=15
  • 2026-10-03T00:36:22+00:00 step 12: query_logs service=orders, minutes=15, limit=50, query=fields @timestamp, @message | sort @timestamp ...
  • 2026-10-03T00:36:22+00:00 step 13: get_metrics minutes=15, dimensions=FunctionName=nightshift-orders, namespace=AWS/Lambda, statistic=Sum...
  • 2026-10-03T00:36:24+00:00 step 14: list_recent_deployments service=orders, minutes=10 (found nothing)
  • 2026-10-03T00:36:25+00:00 step 15: query_logs service=orders, minutes=10, limit=20, query=fields @timestamp, @message | filter level = "... (found nothing)
  • 2026-10-03T00:36:25+00:00 step 16: get_traces minutes=10, service=orders
  • 2026-10-03T00:36:27+00:00 step 17: note_hypotheses hypotheses=likely: orders service has intermittent errors or timeouts causing Lambda error...
  • 2026-10-03T00:36:27+00:00 step 18: finish_investigation hypotheses=likely: orders service has intermittent errors or timeouts causing Lambda error...

Root cause evidence

  • Step 10, get_metrics: {"metric":"AWS/Lambda Errors {} Sum per 60s","window_minutes":15,"points":[["00:31:00Z",0.0],["00:32:00Z",0.0],["00:33:00Z",0.0],["00:34:00Z",53.0],["00:35:00Z",50.0]],"summary":{"min":0.0,"max":53.0,"count":5}}
  • Step 16, get_traces: {"service":"nightshift-orders","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":295,"with_error":125,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.307,"p99":2.872,"max":4.361},"slowest...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Investigate orders service function logs more closely or check downstream dependencies.

Hypotheses

  • likely: orders service has intermittent errors or timeouts causing Lambda errors
  • ruled_out: orders recent deployment caused errors (no recent deployments)
  • ruled_out: Lambda throttling caused errors (no throttles)

Dropped along the way:

  • orders service has a bug or configuration error causing Lambda errors

What the investigation cost

  • 6 model calls, 24,845 input and 791 output tokens (25,636 total)
  • 18 steps, 17 s of wall clock
  • 1,266,756 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments