Skip to the watch report
← All concepts Concept A · Vigil Demo state Calm fleet Incident & recovery Demonstration data — not real equipment
MainsKeeper
Wed 30 Sep

Watch report · four sites, six UPSs, five collectors

Two sites need attention. Broadwater’s rack UPS has been on battery for 13 minutes; Shop has been silent for 12.

Duplex is recovering — contact came back at 15:02 and the history it kept while offline is still uploading. Home is steady.

All four sites are reporting. Every UPS is on utility; nothing needs attention.

The last event was Home’s weekly summary, sent at 09:00. Yesterday’s Duplex outage is resolved and kept in the history.

Reporting
3 of 5expected collectors reporting
5 of 6UPSs communicating with their collectors
3incidents open
1,248history records waiting (Duplex)
5 of 5expected collectors reporting
6 of 6UPSs communicating with their collectors
0incidents open
0history records waiting

Incidents

3 open · newest first by severityNone open

last event

No incidents are open. The most recent event on the watch was Home’s weekly summary, sent at 09:00. Yesterday’s outage at Duplex was resolved and remains in its history.

13 min ago

Critical · Broadwater

BW Rack UPS on battery

The APC Smart-UPS SMT1500 in the rack has been observed on battery for 13 minutes, on live readings 4 s old. Charge is 71% and the UPS reports 0 V at its input. It estimates 18 minutes of runtime remaining1 — a prediction, not an observation.

Acknowledgment
Not acknowledged
Recovery
Not recovered
Maintenance
Not in maintenance
Notifications
Email + push delivered 15:09:14

12 min ago

Warning · Shop

Contact lost with 2 of 2 collectors — cause unknown

Neither shop-bench-win nor shop-esp32-probe has reported since 15:10. We know only that contact stopped; we cannot tell whether power, the network or the machines themselves are responsible.2

Acknowledgment
Not acknowledged
Recovery
Not recovered
Maintenance
Not in maintenance
Notifications
Email delivered 15:15:03
Push failed — retrying
opened 15:02 recovered

Recovering · Duplex

Contact restored; history uploading

Current readings are fresh (3 s). The collector kept a local record while it was out of contact; 1,248 of those records are still on their way. The incident stays open until that history is verified.

Acknowledgment
Acknowledged by jp 13:52
Recovery
Recovered 15:02
Still open
Until backlog verified
Notifications
Delivered

The watch line · last 24 hours

Events at the time they happened

Watch line, 15:22 Tuesday to 15:22 Wednesday, one row per site Broadwater on utility until 15:09, then on battery, with the UPS estimate of 18 minutes drawn dashed beyond now. Home steady. Shop steady until contact was lost at 15:10; no records since. Duplex: probe failures from 13:41, on battery from 14:03 (record received late at 15:17) until the last record at 14:30, no records 14:30 to 14:52, collector restarted at 14:52 plus or minus 90 seconds, contact resumed 15:02. EARLIER 22 HOURS · COMPRESSED LAST 2 HOURS · ENLARGED ×12 AHEAD Tue 15:22 00:00 06:00 12:00 13:30 14:00 14:30 15:00 now Broadwater ▲ on battery ● on utility 15:09 on battery · 13 min observed est. 18 min Home ● steady Shop ○ contact lost ● on utility 15:10 contact lost · cause unknown · last known → No records Duplex ◐ recovering ● on utility ← yesterday’s outage (resolved) sits before this window; see history 13:44 incident opened · 13:52 acknowledged by jp (still open) No records 14:30–14:52 13:41 probe failures begin 14:52 ±90 s · on utility 14:03 on battery (hollow: received late) · at least 27 min; endpoint unknown 15:02 contact resumed · ▽ 15:17 the 14:03 record arrives
Measured, on utility Measured, on battery Estimate (est.) Delayed record, marked with receipt time No records Recorded locally, still uploading Uncertain time

Sites · the log

Written from the latest readings

on battery steady

Broadwater

One UPS on battery · one on utility

Both UPSs on utility

Collector bw-pi-01 · Raspberry Pi 4 · UPS + Connectivity · contact 4 s ago

BW Rack UPS · APC Smart-UPS SMT1500 · protects bw-router, bw-nas-01, bw-switch-24

Status▲ On battery
On battery, observed13 min
Charge71%
Runtime118 min
Load38%
Input0 V
Battery temp31 °C
Reading age4 s
Status● On utility
Charge100%
Runtime141 min
Load36%
Input120 V
Battery temp29 °C
Reading age4 s

BW Office UPS · Eaton 5S 700 · separate circuit · protects bw-desk-ws (no agent)

Status● On utility
Charge100%
Runtime2Not reported
Load12%
TemperatureNot reported
Battery healthNot reported
Reading age4 s

bw-desk-ws is asleep. A sleeping workstation is not an outage and opens no incident.

last heard steady

Shop

Contact lost (cause unknown) · 12 min

Reporting · on utility · fresh

Both designated collectors went quiet at 15:10: shop-bench-win (Windows 11, UPS role) was last heard at 15:10:02, and shop-esp32-probe (ESP32, qualified; connectivity role) at 15:09:58. Nothing since. What follows is the state as it stood then.

Collectors shop-bench-win · Windows 11 · UPS — and shop-esp32-probe · ESP32 (qualified) · Connectivity — both in contact.

Shop Bench UPS · APC Back-UPS Pro 1500 · protects shop-bench-win, shop-cnc-ctrl (no agent), shop-switch

Last known 15:10 · 12 min old3

Status, last known● On utility
Charge100%
Runtime26 min
Load31%
UPS communicationNone since 15:10
Collector contact15:10:02
Reading age12 min
Status● On utility
Charge100%
Runtime26 min
Load31%
UPS communicationCurrent
Collector contactCurrent
Reading ageFresh
contact resumed steady

Duplex

Recovering · uploading history

Reporting · no backlog

Contact resumed at 15:02 after an interruption that began at 13:41. What the collector sees now reaches us within seconds; what it recorded while cut off is arriving behind it. Between 14:30 and 14:52 there is nothing to arrive — the collector was offline and recorded nothing.

Reporting normally. Yesterday’s outage was resolved and is kept in the Duplex history.

Collector duplex-nuc · Fedora Linux 42 · UPS + Connectivity · contact 3 s ago

Duplex Utility UPS · Tripp Lite SMART1500LCD · protects duplex-nuc, duplex-router, duplex-camera-nvr

Status● On utility
Charge78% ↑100%
Runtime29 min
Load27%
Input119 V
TemperatureNot reported
Reading age3 s

↑ recharging after this afternoon’s episode on battery.

History upload · 382 of 1,630 records received · 1,248 pending · oldest 13:41

steady

Home

Both UPSs on utility · both collectors reporting

Collector home-studio · macOS · UPS + Connectivity · contact 6 s ago

Collector home-esp32-garage · ESP32 (qualified) · UPS only · contact 11 s ago

Office Desk UPS · CyberPower CP1500PFCLCD · protects home-studio, home-modem, home-ap

Status● On utility
Charge100%
Runtime42 min
Load22%
Input121 V
TemperatureNot reported
Reading age6 s

Garage UPS · APC Back-UPS 600 · protects garage-door-hub, garage-camera

Status● On utility
Charge100%
RuntimeNot reported
LoadNot reported
InputNot reported
TemperatureNot reported
Reading age11 s

Over the last 30 days, observed internet availability was 99.93%, across 98.1% coverage — 13 h 41 m of the period is unknown and not counted either way. Reporting availability for the same 30 days was 99.4%.4