# 2026-04-28 — bulletproof recovery + FWS / fault polling + EG4 discovery refresh

Mostly resilience work today, prompted by waking up to find both powermon
services inactive and no PV / battery data flowing from the inverters since
~06:11. Root-caused, recovered, and hardened the failure mode so it can't
strand the stack again. While in there: closed two related polling gaps
(faults weren't being captured at all, and EG4 discovery configs had
fallen out of the broker — InfluxDB exporter had stopped seeing pack data).

## What happened

1. **Morning incident: powermon offline since 06:11.** udev fired the
   resolver during what looks like a sunrise cold-start window — both
   inverters were briefly unresponsive on USB-HID (`/dev/hidraw0`
   returned no PI18 reply; `/dev/hidraw1` returned 22 zero bytes for the
   serial query). Resolver hit its 10 s internal retry, gave up, exited
   with code 2. Powermon services had `Requires=lvx-resolve-links` so
   systemd refused to start them, and there was no retry path — stack
   stayed dead until manual intervention ~10 h later.

2. **Recovery.** Re-ran the resolver manually; both inverters responded
   normally on the second attempt. Started powermon services; PV
   voltages (305.2 V / 307.2 V), battery voltages (54.0 V), and all the
   usual telemetry flowed back within seconds.

3. **Bulletproofing the resolver path.** Three coordinated changes so a
   transient morning blip can't latch the whole stack offline:

   a. **Resolver → always-exit-0.** Best-effort policy: write whatever
      symlinks it can resolve, log warnings for the missing ones,
      always return success. Only-relink-on-change semantics so a
      no-op timer-driven run doesn't churn the symlinks. When symlinks
      *do* move (post-power-cycle), resolver auto-restarts powermon
      so its open hidraw fds re-bind.

   b. **`Wants=` instead of `Requires=`** on both powermon services.
      Resolver failure no longer propagates as "permanently failed"
      to dependent units. Powermon's own `Restart=always` handles the
      retry loop until `/dev/lvx6048-{1,2}` appear.

   c. **Periodic systemd timer** for the resolver
      (`lvx-resolve-links.timer`) — `OnBootSec=2min`,
      `OnUnitActiveSec=5min`. Belt-and-suspenders against any
      hot-plug case where udev didn't fire (stuck-USB scenario like
      this morning's).

   d. **`StartLimitIntervalSec=0`** on both powermon services — disables
      systemd's default "stop trying after 5 fails in 10 s" rate-limiter,
      so the retry loop continues indefinitely until success.

   Worst-case data gap is now ~2 min (timer cycle + couple polls) vs.
   the previous "until a human notices and runs systemctl restart".

4. **FWS fault polling added to powermon.** A separate gap surfaced
   during the morning's troubleshooting: front-panel showed a "32"
   indicator, and we discovered our `powermon.yaml` was polling
   `GS / MOD / PIRI / ET` only — no `FWS`. Faults / warnings weren't
   reaching MQTT at all. Added FWS (every 30 s) to both inverter
   configs.

5. **Defensive FWS fault-code decoder.** Same shape as the MOD `06`/`07`
   crashes from earlier in the week — an unknown fault code would
   `KeyError` and crash powermon. Pre-populated the OPTION dict with
   placeholder labels for codes 00–99, then layered the known names on
   top. Future unknown codes show as `"Fault 32"` rather than crashing.
   Also unblocks adding the four PI18 setters (PE / PD / POPM / PF) to
   the shim later — same pattern.

6. **About the LCD's "32".** Direct FWS queries to both units returned
   `fault_code=No fault` while `pv_loss_warning=enabled`. The LCD was
   most likely showing the PV-loss warning code (no F/W prefix on this
   firmware's display), correlating with the cold-start window. Not a
   real fault — but worth re-confirming the LCD format next time so we
   know whether 32-with-prefix is a thing on this hardware.

7. **EG4 lifepower discovery configs vanished.** Independent finding:
   when checking why `lvx6048_*_fault_code` wasn't appearing in HA, I
   noticed zero retained `homeassistant/sensor/lifepower4_*/config`
   topics on the broker. HA had cached entity registrations and was
   showing live state, but the canonical discovery configs were gone —
   likely cleared by an earlier `eg4-purge-orphans` invocation or
   broker retention loss. InfluxDB exporter relies on those configs to
   know what entities to publish, so pack voltages / SoCs had stopped
   reaching InfluxDB.

   Fix: `systemctl restart eg4-battery.service`. Daemon re-publishes
   all 207 retained discovery configs at startup (3 packs × ~69
   entities). HA refreshes its registry, InfluxDB resumes capturing.

## Net effect

| Layer | Before | After |
|---|---|---|
| Morning-blip recovery | manual `systemctl restart` (~10 h gap) | automatic ~2 min gap |
| Resolver exit on missing serial | code 2 (systemd marks unit failed) | always 0 (best-effort) |
| Resolver runs | only on udev hot-plug | + every 5 min via timer |
| Powermon ↔ resolver coupling | `Requires=` (failure cascades) | `Wants=` (independent retry) |
| Powermon retry budget | ~5 attempts (default StartLimit) | unbounded (`StartLimitIntervalSec=0`) |
| Fault-code → MQTT | not polled | FWS every 30 s |
| Unknown fault code | `KeyError` crash | shows as `"Fault NN"` |
| EG4 discovery configs | missing (purged) | 207 retained, InfluxDB unblocked |

## Files touched

```
M  LVX6048/bin/lvx-resolve-links                                    (always-exit-0; relink-on-change; auto-restart powermon if symlinks move)
M  LVX6048/config/powermon/powermon.yaml                             (added FWS poll @ 30s)
M  LVX6048/config/powermon/powermon2.yaml                            (same)
M  LVX6048/etc/systemd/system/powermon.service                       (StartLimitIntervalSec=0)
M  LVX6048/etc/systemd/system/powermon.service.d/10-resolver.conf    (Requires= -> Wants=)
M  LVX6048/etc/systemd/system/powermon2.service                      (same)
M  LVX6048/etc/systemd/system/powermon2.service.d/10-resolver.conf   (same)
M  LVX6048/install.sh                                                 (installs the timer)
M  LVX6048/powermon-patches/pi18.py                                   (defensive fault-code decoder, 00-99)
A  LVX6048/etc/systemd/system/lvx-resolve-links.timer                 (every 5 min)
A  LVX6048/2026-04-28.md                                              (this file)
```

## How to verify

```bash
# Timer running?
systemctl list-timers lvx-resolve-links.timer

# Resolver idempotent? (no powermon bounce on no-change)
sudo systemctl restart lvx-resolve-links.service
journalctl -u lvx-resolve-links.service -n 8 --no-pager
# Expect: "(unchanged)" lines, no "restarting powermon" line.

# Wants= confirmed?
systemctl show powermon.service -p Wants -p Requires
# Expect: Wants=...lvx-resolve-links.service... ; Requires= line should NOT
# include lvx-resolve-links.

# Synthetic morning-blip test (optional, disruptive):
sudo rm /dev/lvx6048-{1,2}
sudo systemctl restart powermon.service
# Expect: powermon enters Restart loop; within 5 min the timer fires the
# resolver, recreates symlinks, powermon's next retry succeeds.

# Fault polling alive
mosquitto_sub -h <broker> -u mqtt -P <pass> -W 35 -v \
    -t 'homeassistant/sensor/lvx6048_1_fault_code/state' \
    -t 'homeassistant/sensor/lvx6048_2_fault_code/state'
# Expect: "No fault" published every 30 s on each unit.
```

## Loose ends / next pass

- **FWS warning booleans aren't getting individual HA entities.** The 16
  `enabled/disabled` flags (PV loss, over-temp, fan locked, …) are
  decoded by powermon's screen formatter but the hass formatter filters
  them because they lack `unit_of_measurement` / `device_class` /
  `reading_type` metadata. Worth a separate pass to expose them as
  `binary_sensor` entities so HA gets a real "PV loss" indicator
  alongside `fault_code`.
- **LCD code format confirmation.** Next time you can look at the LCD,
  check whether warning codes display as bare "32", "W32", or
  something else — informs whether to add a friendly mapping for
  warning codes in the docs.
- **Faster timer cadence?** Currently 5 min (worst-case ~2 min gap).
  Could drop to 1–2 min if morning blips become more frequent. Trade-
  off: more steady-state resolver runs (each costs one PI18 ID query
  per inverter, harmless but visible in logs).
- **eg4-battery discovery-config retention.** The configs DID get
  purged at some point — not certain what triggered it. Next
  investigation: confirm whether `eg4-purge-orphans` is over-eager,
  whether the broker is dropping retains, or whether a daemon restart
  is needed. Worth a `mosquitto_sub` watch on the broker for retain
  changes.