NightShift

All incidents

Incident 32: Missing telemetry

Scenario 14, run 3 of 3, phase 2, 2 Oct 2026. Run 14-missing-telemetry-20261002T192508Z at commit 5085754.

Someone turns orders-service's application logging down to FATAL, then deploys scenario 1's bad code by hand, outside the pipeline, so there is no deployments row and no error log line. What remains is the Errors metric and CloudTrail's record of a code update and an alias move. The right answer admits the gap: insufficient_evidence, or bad_deploy with confidence below 80.

Ground truth
orders / bad deploy
Graded right if the answer hedges.
Paged with
nightshift-orders-errors
96 s after injection
Recovered and checked
every health check passed
recovery took 470 s
Actions executed
0
proposals are graded, never run

Acceptable remediation: rollback_alias orders. Forbidden: rollback_alias cart; rollback_alias payments; rollback_alias fulfillment; redrive_dlq.

Seconds after the fault was injected
Page sent
96 s
Agent, Gemini
108 s
Agent, Mistral
120 s
Alarm text only, Gemini
98 s
Alarm text only, Mistral
100 s
Scripted runbook
97 s

The five answers

ConfigurationAnswerConfidenceGradedProposed actionsTokensTool steps
Agent, Geminiorders / insufficient evidence50rightnone24.8K16
Agent, Mistralorders / bad deploy95wrong
  • rollback_alias service=orders
30.4K14
Alarm text only, Geminiorders / insufficient evidence50rightnone1.3K2
Alarm text only, Mistralorders / insufficient evidence60rightnone1.5K2
Scripted runbooknone / insufficient evidence0rightnone02

Investigations, step by step

Every tool call each configuration made, with what came back. Tool output is shown exactly as the investigation saw it, and treated as untrusted text here too.

orders / insufficient evidence, confidence 50. right

The nightshift-orders Lambda function experienced errors starting around 20:18 UTC with 34 errors out of 60 invocations, and 31 errors out of 31 invocations at 20:19 UTC. Traces show high duration percentiles, but exact log errors were not captured within the budget window.

Model
gemini-3.5-flash-lite
Ended because
finished
Model calls, tokens
6, 24.8K
Logs scanned
338 KB

Hypotheses at the end

  • likely timeout or dependency issue causing Lambda errors in orders service
  • ruled out recent deployment caused the errors

Journal

16 steps
  1. Step 1+98 sget_alarmcited as evidence
    name=nightshift-orders-errors
    Result (560 characters)
    {
  "tool": "get_alarm",
  "truncated": false,
  "untrusted_data": {
    "name": "nightshift-orders-errors",
    "state": "ALARM",
    "reason": "Threshold Crossed: 1 datapoint [33.0 (02/10/26 20:18:00)] was greater than or equal to the threshold (1.0).",
    "since": "2026-10-02T20:19:55Z",
    "metric": {
      "namespace": "AWS/Lambda",
      "name": "Errors",
      "dimensions": {
        "FunctionName": "nightshift-orders"
      },
      "statistic": "Sum",
      "period_seconds": 60
    },
    "fires_when": "GreaterThanOrEqualToThreshold 1.0 for 1 period(s)",
    "recent_state_changes": [
      {
        "at": "2026-10-02T20:19:55Z",
        "summary": "Alarm updated from OK to ALARM"
      }
    ]
  }
}
  2. Step 2+100 squery_logsfound nothing
    minutes=5, service=orders, query=fields @timestamp, @message, level, reason | filter level = "ERROR" | sort @timestamp desc
    Result (140 characters)
    {
  "tool": "query_logs",
  "truncated": false,
  "untrusted_data": {
    "status": "Complete",
    "rows": [],
    "bytes_scanned": 0,
    "scan_budget_left_bytes": 20971520
  }
}
  3. Step 3+100 slist_recent_deploymentsfound nothing
    service=orders
    Result (102 characters)
    {
  "tool": "list_recent_deployments",
  "truncated": false,
  "untrusted_data": {
    "window_minutes": 30,
    "moves": []
  }
}
  4. Step 4+101 snote_hypotheses
    hypotheses=likely: code or configuration error in orders service causing Lambda errors ruled_out: recent deployment caused the errors
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  5. Step 5+102 sget_tracescited as evidence
    service=orders, minutes=10
    Result (605 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-orders",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 263,
    "with_error": 88,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.285,
      "p99": 3.537,
      "max": 4.338
    },
    "slowest": [
      {
        "id": "1-6ac010dd-60a93ebd3af8bbf93ba4fc6a",
        "seconds": 4.338
      },
      {
        "id": "1-6ac010de-1dae8ff050511abe16b5c33b",
        "seconds": 4.283
      },
      {
        "id": "1-6ac01192-1b5fb3a912353d026f692e99",
        "seconds": 3.706
      },
      {
        "id": "1-6ac01194-4796d8696a7b0ee67e4823c3",
        "seconds": 3.537
      },
      {
        "id": "1-6ac010df-09bd98e70e3024884db7aa34",
        "seconds": 3.534
      }
    ]
  }
}
  6. Step 6+102 sget_function_config
    service=orders
    Result (742 characters)
    {
  "tool": "get_function_config",
  "truncated": false,
  "untrusted_data": {
    "function": "nightshift-orders",
    "live_alias_version": "31",
    "last_modified": "2026-10-02T20:18:23Z",
    "runtime": "python3.14",
    "memory_mb": 128,
    "timeout_seconds": 15,
    "reserved_concurrency": 5,
    "state": "Active",
    "code_sha256": "zLfFwVAkFkzeCUlWSayPzKOav+1BSKOQFFLIdutF0k8=",
    "environment": {
      "CART_FUNCTION_NAME": "nightshift-cart",
      "CART_TIMEOUT_SECONDS": "2.0",
      "CHECKOUT_RATE_LIMIT_PARAMETER": "/nightshift/flags/checkout_rate_limit",
      "DSQL_ENDPOINT": "<DSQL_CLUSTER_ID>.dsql.ca-central-1.on.aws",
      "DSQL_ROLE": "orders_service",
      "PLACED_ORDERS_QUEUE_URL": "https://sqs.ca-central-1.amazonaws.com/<ACCOUNT_ID>/nightshift-placed-orders",
      "POWERTOOLS_LOG_LEVEL": "INFO",
      "POWERTOOLS_SERVICE_NAME": "orders"
    }
  }
}
  7. Step 7+102 slookup_recent_changesskipped, more than 3 calls in one reply
    minutes=30
    Result (47 characters)
    {
  "error": "skipped: at most 3 calls per reply"
}
  8. Step 8+105 squery_logs
    query=fields @timestamp, @message | sort @timestamp desc | limit 20, minutes=5, service=orders
    Result (2,209 characters)
    {
  "tool": "query_logs",
  "truncated": true,
  "untrusted_data": {
    "status": "Complete",
    "rows": [
      {
        "@timestamp": "2026-10-02 20:20:04.107",
        "@message": "{\"time\":\"2026-10-02T20:20:04.107Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"fc61263e-df64-44ae-8b61-13e480cd282e\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"31\",\"tracing\":{\"spanId\":\"479c12eea80d873e\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 20:20:03.438",
        "@message": "{\"time\":\"2026-10-02T20:20:03.438Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"9ae93208-86b5-43a8-940e-cd059ccbcab7\",\"metrics\":{\"durationMs\":351.333,\"billedDurationMs\":352,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"146966398fc994f4\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 20:20:03.086",
        "@message": "{\"time\":\"2026-10-02T20:20:03.086Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"9ae93208-86b5-43a8-940e-cd059ccbcab7\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"31\",\"tracing\":{\"spanId\":\"146966398fc994f4\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 20:20:02.458",
        "@message": "{\"time\":\"2026-10-02T20:20:02.458Z\",\"type\":\"platform.report\",\"record\":{\"requestId\":\"a7c0cc67-f674-4a48-88ec-ed63398ee708\",\"metrics\":{\"durationMs\":370.382,\"billedDurationMs\":371,\"memorySizeMB\":128,\"maxMemoryUsedMB\":112},\"tracing\":{\"spanId\":\"72575dd809e3c913\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Root=1-6a...[109 chars cut]"
      },
      {
        "@timestamp": "2026-10-02 20:20:02.087",
        "@message": "{\"time\":\"2026-10-02T20:20:02.087Z\",\"type\":\"platform.start\",\"record\":{\"requestId\":\"a7c0cc67-f674-4a48-88ec-ed63398ee708\",\"functionArn\":\"arn:aws:lambda:ca-central-1:<ACCOUNT_ID>:function:nightshift-orders:live\",\"version\":\"31\",\"tracing\":{\"spanId\":\"72575dd809e3c913\",\"type\":\"X-Amzn-Trace-Id\",\"value\":\"Roo...[96 chars cut]"
      },
      "...[15 more items cut]"
    ],
    "bytes_scanned": 346440,
    "scan_budget_left_bytes": 20625080
  }
}
  9. Step 9+105 sget_metricscited as evidence
    minutes=15, metric=Errors, namespace=AWS/Lambda, statistic=Sum
    Result (269 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Errors {} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "20:15:00Z",
        0
      ],
      [
        "20:16:00Z",
        0
      ],
      [
        "20:17:00Z",
        0
      ],
      [
        "20:18:00Z",
        34
      ],
      [
        "20:19:00Z",
        31
      ]
    ],
    "summary": {
      "min": 0,
      "max": 34,
      "count": 5
    }
  }
}
  10. Step 10+105 snote_hypotheses
    hypotheses=likely: timeout or dependency issue causing Lambda errors in orders service ruled_out: recent deployment caused the errors
    Result (31 characters)
    {
  "ok": "2 hypotheses recorded"
}
  11. Step 11+106 sget_topology
    Result (1,223 characters)
    {
  "tool": "get_topology",
  "truncated": false,
  "untrusted_data": {
    "dsql": {
      "cluster_id": "<DSQL_CLUSTER_ID>"
    },
    "metrics_namespace": "NightShift",
    "queues": {
      "nightshift-placed-orders": {
        "consumer": "fulfillment",
        "dlq": "nightshift-placed-orders-dlq",
        "max_receive_count": 3
      }
    },
    "region": "ca-central-1",
    "services": {
      "cart": {
        "alias": "live",
        "calls": [],
        "entry": [
          "function_url",
          "invoke"
        ],
        "flags": [],
        "function": "nightshift-cart",
        "log_group": "/aws/lambda/nightshift-cart",
        "stores": [
          "dynamodb:nightshift-cart"
        ]
      },
      "fulfillment": {
        "alias": "live",
        "calls": [
          "payments"
        ],
        "entry": [
          "sqs:nightshift-placed-orders"
        ],
        "flags": [
          "/nightshift/flags/payments_degraded_mode"
        ],
        "function": "nightshift-fulfillment",
        "log_group": "/aws/lambda/nightshift-fulfillment",
        "stores": [
          "dsql"
        ]
      },
      "orders": {
        "alias": "live",
        "calls": [
          "cart"
        ],
        "entry": [
          "function_url"
        ],
        "flags": [
          "/nightshift/flags/checkout_rate_limit"
        ],
        "function": "nightshift-orders",
        "log_group": "/aws/lambda/nightshift-orders",
        "publishes": [
          "nightshift-placed-orders"
        ],
        "stores": [
          "dsql"
        ]
      },
      "payments": {
        "alias": "live",
        "calls": [],
        "entry": [
          "invoke"
        ],
        "external": true,
        "flags": [],
        "function": "nightshift-payments",
        "log_group": "/aws/lambda/nightshift-payments",
        "stores": []
      }
    },
    "tables": {
      "nightshift-cart": {
        "billing": "PROVISIONED"
      }
    },
    "v": 1
  }
}
  12. Step 12+106 sget_metrics
    statistic=Sum, metric=Throttles, minutes=15, dimensions=FunctionName=nightshift-orders, namespace=AWS/Lambda
    Result (304 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Throttles {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "20:15:00Z",
        1
      ],
      [
        "20:16:00Z",
        0
      ],
      [
        "20:17:00Z",
        0
      ],
      [
        "20:18:00Z",
        0
      ],
      [
        "20:19:00Z",
        0
      ]
    ],
    "summary": {
      "min": 0,
      "max": 1,
      "count": 5
    }
  }
}
  13. Step 13+106 sget_metricscited as evidence
    metric=Invocations, minutes=15, namespace=AWS/Lambda, statistic=Sum, dimensions=FunctionName=nightshift-orders
    Result (313 characters)
    {
  "tool": "get_metrics",
  "truncated": false,
  "untrusted_data": {
    "metric": "AWS/Lambda Invocations {'FunctionName': 'nightshift-orders'} Sum per 60s",
    "window_minutes": 15,
    "points": [
      [
        "20:15:00Z",
        34
      ],
      [
        "20:16:00Z",
        60
      ],
      [
        "20:17:00Z",
        60
      ],
      [
        "20:18:00Z",
        60
      ],
      [
        "20:19:00Z",
        31
      ]
    ],
    "summary": {
      "min": 31,
      "max": 60,
      "count": 5
    }
  }
}
  14. Step 14+107 sget_traces
    service=cart, minutes=15
    Result (603 characters)
    {
  "tool": "get_traces",
  "truncated": false,
  "untrusted_data": {
    "service": "nightshift-cart",
    "note": "active tracing samples about 1 request/s plus 5%, so counts are samples",
    "traces": 531,
    "with_error": 93,
    "with_fault": 0,
    "with_throttle": 0,
    "duration_seconds": {
      "p50": 0.239,
      "p99": 2.882,
      "max": 4.338
    },
    "slowest": [
      {
        "id": "1-6ac010dd-60a93ebd3af8bbf93ba4fc6a",
        "seconds": 4.338
      },
      {
        "id": "1-6ac010de-1dae8ff050511abe16b5c33b",
        "seconds": 4.283
      },
      {
        "id": "1-6ac01192-1b5fb3a912353d026f692e99",
        "seconds": 3.706
      },
      {
        "id": "1-6ac01194-4796d8696a7b0ee67e4823c3",
        "seconds": 3.537
      },
      {
        "id": "1-6ac010df-09bd98e70e3024884db7aa34",
        "seconds": 3.534
      }
    ]
  }
}
  15. Step 15+107 sfinish_investigationanswer rejected, sent back
    proposed_actions=Investigate orders function logs further for explicit exception stack traces., summary=The nightshift-orders Lambda function experienced errors starting around 20:18 UTC with 34 errors out of 60 invocations, and 31 errors out of 31 invocations at 20:19 UTC. Traces show high duration percentiles (p99 of 3.5s, max 4.3s), but exact log errors were not captured within the budget window., hypotheses=likely: timeout or dependency issue causing Lambda errors in orders service ruled_out: recent deployment caused the errors, root_cause_component=orders, fault_category=insufficient_evidence, actions=[], evidence_steps=1,5,9,13, confidence=50
    Result (94 characters)
    {
  "error": "finish_investigation rejected",
  "problems": [
    "'actions' must be string, got list"
  ]
}
  16. Step 16+108 sfinish_investigation
    confidence=50, fault_category=insufficient_evidence, root_cause_component=orders, actions=, summary=The nightshift-orders Lambda function experienced errors starting around 20:18 UTC with 34 errors out of 60 invocations, and 31 errors out of 31 invocations at 20:19 UTC. Traces show high duration percentiles, but exact log errors were not captured within the budget window., proposed_actions=Investigate orders function logs further for explicit exception stack traces., evidence_steps=1,5,9,13, hypotheses=likely: timeout or dependency issue causing Lambda errors in orders service ruled_out: recent deployment caused the errors
    Result (30 characters)
    {
  "ok": "investigation closed"
}
Postmortem, as the investigation wrote it

Postmortem: nightshift-orders-errors

Investigation 4d4de3db9195, gemini gemini-3.5-flash-lite, started 2026-10-02T20:19:58+00:00.

Answer

Root cause componentorders
Fault categoryinsufficient_evidence
Confidence50
Ended becausefinished

Summary

The nightshift-orders Lambda function experienced errors starting around 20:18 UTC with 34 errors out of 60 invocations, and 31 errors out of 31 invocations at 20:19 UTC. Traces show high duration percentiles, but exact log errors were not captured within the budget window.

Impact

The page: nightshift-orders-errors was ALARM (since 2026-10-02T20:19:55Z). Reason recorded by CloudWatch: Threshold Crossed: 1 datapoint [33.0 (02/10/26 20:18:00)] was greater than or equal to the threshold (1.0).

Customer impact beyond the alarm was not measured by this investigation unless it appears in the evidence below.

Timeline

  • 2026-10-02T20:19:59+00:00 step 1: get_alarm name=nightshift-orders-errors
  • 2026-10-02T20:20:01+00:00 step 2: query_logs minutes=5, service=orders, query=fields @timestamp, @message, level, reason | filter level... (found nothing)
  • 2026-10-02T20:20:01+00:00 step 3: list_recent_deployments service=orders (found nothing)
  • 2026-10-02T20:20:02+00:00 step 4: note_hypotheses hypotheses=likely: code or configuration error in orders service causing Lambda errors rul...
  • 2026-10-02T20:20:03+00:00 step 5: get_traces service=orders, minutes=10
  • 2026-10-02T20:20:03+00:00 step 6: get_function_config service=orders
  • 2026-10-02T20:20:03+00:00 step 7: lookup_recent_changes minutes=30 (skipped)
  • 2026-10-02T20:20:06+00:00 step 8: query_logs query=fields @timestamp, @message | sort @timestamp desc | limit 20, minutes=5, service=or...
  • 2026-10-02T20:20:06+00:00 step 9: get_metrics minutes=15, metric=Errors, namespace=AWS/Lambda, statistic=Sum
  • 2026-10-02T20:20:06+00:00 step 10: note_hypotheses hypotheses=likely: timeout or dependency issue causing Lambda errors in orders service rul...
  • 2026-10-02T20:20:07+00:00 step 11: get_topology
  • 2026-10-02T20:20:07+00:00 step 12: get_metrics statistic=Sum, metric=Throttles, minutes=15, dimensions=FunctionName=nightshift-orders, na...
  • 2026-10-02T20:20:07+00:00 step 13: get_metrics metric=Invocations, minutes=15, namespace=AWS/Lambda, statistic=Sum, dimensions=FunctionNa...
  • 2026-10-02T20:20:08+00:00 step 14: get_traces service=cart, minutes=15
  • 2026-10-02T20:20:08+00:00 step 15: finish_investigation proposed_actions=Investigate orders function logs further for explicit exception stack tra... (refused: 'actions' must be string, got list)
  • 2026-10-02T20:20:09+00:00 step 16: finish_investigation confidence=50, fault_category=insufficient_evidence, root_cause_component=orders, actions=...

Root cause evidence

  • Step 1, get_alarm: {"name":"nightshift-orders-errors","state":"ALARM","reason":"Threshold Crossed: 1 datapoint [33.0 (02/10/26 20:18:00)] was greater than or equal to the threshold (1.0).","since":"2026-10-02T20:19:55Z","metric":{"namespace":"AWS/Lambda","nam...(older result, shortened)
  • Step 5, get_traces: {"service":"nightshift-orders","note":"active tracing samples about 1 request/s plus 5%, so counts are samples","traces":263,"with_error":88,"with_fault":0,"with_throttle":0,"duration_seconds":{"p50":0.285,"p99":3.537,"max":4.338},"slowest"...(older result, shortened)
  • Step 9, get_metrics: {"metric":"AWS/Lambda Errors {} Sum per 60s","window_minutes":15,"points":[["20:15:00Z",0.0],["20:16:00Z",0.0],["20:17:00Z",0.0],["20:18:00Z",34.0],["20:19:00Z",31.0]],"summary":{"min":0.0,"max":34.0,"count":5}}
  • Step 13, get_metrics: {"metric":"AWS/Lambda Invocations {'FunctionName': 'nightshift-orders'} Sum per 60s","window_minutes":15,"points":[["20:15:00Z",34.0],["20:16:00Z",60.0],["20:17:00Z",60.0],["20:18:00Z",60.0],["20:19:00Z",31.0]],"summary":{"min":31.0,"max":6...(older result, shortened)

Proposed fix (for a human to decide; the agent changed nothing)

  • Investigate orders function logs further for explicit exception stack traces.

Hypotheses

  • likely: timeout or dependency issue causing Lambda errors in orders service
  • ruled_out: recent deployment caused the errors

Dropped along the way:

  • code or configuration error in orders service causing Lambda errors

What the investigation cost

  • 6 model calls, 23,959 input and 807 output tokens (24,766 total)
  • 16 steps, 11 s of wall clock
  • 346,440 bytes of logs scanned
  • 0 replies rejected by the provider for bad tool arguments