Synthetic walkthrough · one site

Broadwater outage walkthrough

Utility power, the WAN, and the collector do not fail or recover together. This is the only step-by-step incident in the guide. The pilot checklist and Chapter 8 point here.

Synthetic. Clock 2026-04-14, offset illustrated as −07:00. Heartbeat 60 s, so the accepted contact grace is 5 minutes. Charge, load, and runtime are not invented. Nothing here is a measurement from deployed equipment.

Cast

Thing	Name in this story	Role
Site	Broadwater · `broadwater-lan`	One namespace
Collector	`bw-pi-01`	Always-on Pi witness. It keeps running.
UPS	`ups-bw-rack`	The unit that goes on battery, then returns to utility.
Second UPS	`ups-bw-office`	Stays on utility the whole time, including during the silence.

Timeline

What was known at each stage

09:00 · Ordinary operation

Collector: Both UPS sources answer. Rack and office are on utility. Connectivity checks succeed.
Local buffer: The new rows are appended, then uploaded. The acknowledgement cursor can advance.
Server: Current state matches the readings. Contact is live.
Operator: Can treat the readings as current. Nothing is only “last known.”
After replay: Nothing is waiting. This span was accepted live.
Notifications: None.

09:12 · Utility fails. The WAN is still up.

Collector: ups-bw-rack is observed on battery. ups-bw-office stays on utility. The collector process is up.
Local buffer: The on-battery observation is stored with observation time 09:12, then uploaded. Illustrative sequence 4412.
Server: Receives it about two seconds later. Current state for the rack UPS becomes on battery. This is ordinary latency, not replay.
Operator: Sees on battery, with the UPS estimate labelled as an estimate if the UPS reported one. No estimate is invented here.
After replay: This row is already acknowledged. It is not part of the later batch.
Notifications: The on-battery delay is 60 seconds, so nothing fires at 09:12. Below that delay the design records an event only.

09:13 · On-battery alert, while uploads still work

Collector: Still reading the rack UPS on battery. WAN checks still succeed.
Local buffer: New samples continue to upload and ack.
Server: The incident is declared from fresh evidence. First evidence is 09:12. Declaration is 09:13, after the accepted delay.
Operator: Can conclude the rack UPS was on battery. The office UPS was not. Site contact is fine.
After replay: This declaration stays the one alert for that condition.
Notifications: One on-battery notification for ups-bw-rack. No notification for the office UPS.

09:20 · WAN fails. Collection does not.

Collector: Connectivity checks fail. The probe has confirmed a connectivity failure locally. Both UPS reads continue. The collector is not down.
Local buffer: New observations, including ongoing battery operation, stay on disk. Upload fails and will retry. Rows already acked are not kept for resend.
Server: Stops hearing the witness. Inside the grace window the last rack state is still on battery and the office state is still on utility. That is last known, not current.
Operator: Cannot yet call it contact lost. Cannot call the silence a power failure. The local connectivity finding is not on the server yet.
After replay: The buffered span will show the rack still on battery and the office still on utility, plus the failed checks.
Notifications: No new notification. The on-battery incident stays open because nothing fresh says it cleared.

09:25 · Server declares contact lost

Collector: Still sampling. It cannot deliver the local connectivity finding.
Local buffer: Growing. If the bounded buffer overflowed, the dropped span would be a gap. This story assumes it does not overflow.
Server: Contact lost — cause unknown. Grace is max(5 min, 3 × heartbeat). With a 60-second heartbeat the grace is 5 minutes.
Operator: Knows the witness went silent. Does not know from the server alone that the WAN failed, or whether either UPS is still on battery.
After replay: The cause becomes “connectivity failure confirmed by local probe,” because the buffered checks say so. Until then the server must not invent that cause.
Notifications: A contact-lost notification. Not a second power-failure alert derived from the missing heartbeat.

09:41 · Utility returns. The WAN is still down.

Collector: Observes ups-bw-rack back on utility. Office UPS never left utility. Samples stay local.
Local buffer: The return is stored at observation time 09:41. It has no receipt time yet.
Server: Unchanged: contact lost, rack last known on battery, office last known on utility.
Operator: Must not treat last known as current. The screen should say contact lost and show ages, not a live on-battery state.
After replay: History places this transition at 09:41 and ends the observed battery episode. The row does not recover the incident, and it does not set the recovery time.
Notifications: None can be delivered. Replay of this row will not send a recovery notice.

09:48 · WAN returns. Heartbeat, a fresh reading, and replay.

Collector: Sends a fresh heartbeat and a current reading of ups-bw-rack on utility, and uploads the unacknowledged span. Sampling never stopped, so there is no collector-down incident. The fresh reading can arrive while the backlog is still draining.
Local buffer: Replayed rows keep observation time 09:41. Rows retire only after ingestion acknowledges a contiguous prefix. That retirement rule is Proposed. A retried batch is idempotent.
Server: The fresh heartbeat restores collector contact. A separate fresh UPS reading confirms current utility operation and recovers the on-battery incident. Recovery does not wait for history to finish syncing. Replayed rows update history at their observation times, are marked received late, and do not overwrite this newer current state.
Operator: Sees contact restored by the heartbeat, and current utility from the fresh reading. As the drain proceeds, rack history shows on battery from 09:12 until 09:41. The office UPS was on utility throughout. Receipt time explains the late history. It is not the recovery time.
After replay: The unknown span becomes known history because the buffer held it. A dropped span would have stayed a hatched gap. Restoring that history is not operational recovery.
Notifications: No second on-battery alert, and the replayed 09:41 row does not recover or resolve anything. Contact lost recovers on the fresh heartbeat. The on-battery incident recovers on the fresh utility reading. If the rack UPS were still on battery at reconnect, that fresh reading would stay alertable during the drain.

09:53 · Five stable minutes after recovery

Collector: Still reporting utility on the rack UPS and utility on the office UPS.
Local buffer: The replayed prefix can be retired only after the delivery acknowledgement. That does not close an incident.
Server: Each incident resolves on its own clock: five stable minutes after its own recovery. On-battery recovery is the fresh reading near 09:48, so resolution is 09:53. Contact-lost recovery is the 09:48 heartbeat, not the 09:41 observation.
Operator: Can see both incidents resolved. An operator acknowledgement, if anyone made one, remains a separate timestamp.
Notifications: Resolution is not a new alert. Nothing is backdated to 09:41.

Notification ledger

Condition	What happens
Rack on battery	One alert at 09:13. Replay does not repeat it.
Office UPS	No power alert. Replay shows it never left utility.
Contact lost	One alert at 09:25. Recovers on the fresh 09:48 heartbeat, not when the replayed UPS rows arrive.
Historical restoration	Replay writes the 09:41 utility return at its observation time and ends the battery episode in history. It does not recover or resolve the on-battery incident.
Recovery	On-battery recovery is the separate fresh UPS reading near 09:48. It does not wait for the backlog to drain, and it is not backdated to 09:41.
Resolution	Each incident resolves only after five stable minutes following its own recovery. For this morning that is 09:53. Show the receipt time of the 09:41 observation when it explains the late history.
Operator acknowledgement	If someone acknowledges either incident, that timestamp is separate. It does not recover or resolve it.
Delivery acknowledgement	The acknowledgement cursor lets the collector retire a contiguous prefix. That mechanism is Proposed. It is not operator acknowledgement and it does not close an incident.

What is still unknown after a clean replay

Surviving on battery does not certify capacity. Battery Assurance still needs eligible evidence on the current installation.
A full charge or a UPS-reported self-test PASS would not close that question either. This story does not include one.
The office UPS being fine does not describe the rack UPS, and the reverse is also true.
Prometheus scraping bw-pi-01 during the gap would have shown silence, then current gauges. It would not have produced the 09:20–09:48 history.

Interface detail for the upload and the ack is in the buffer boundary and the ingestion boundary. Enrollment of this collector is a different morning: enrollment sequence.