171 lines
8.6 KiB
Markdown
171 lines
8.6 KiB
Markdown
|
# 2026-04-28 — bulletproof recovery + FWS / fault polling + EG4 discovery refresh
|
|||
|
|
|
|||
|
|
Mostly resilience work today, prompted by waking up to find both powermon
|
|||
|
|
services inactive and no PV / battery data flowing from the inverters since
|
|||
|
|
~06:11. Root-caused, recovered, and hardened the failure mode so it can't
|
|||
|
|
strand the stack again. While in there: closed two related polling gaps
|
|||
|
|
(faults weren't being captured at all, and EG4 discovery configs had
|
|||
|
|
fallen out of the broker — InfluxDB exporter had stopped seeing pack data).
|
|||
|
|
|
|||
|
|
## What happened
|
|||
|
|
|
|||
|
|
1. **Morning incident: powermon offline since 06:11.** udev fired the
|
|||
|
|
resolver during what looks like a sunrise cold-start window — both
|
|||
|
|
inverters were briefly unresponsive on USB-HID (`/dev/hidraw0`
|
|||
|
|
returned no PI18 reply; `/dev/hidraw1` returned 22 zero bytes for the
|
|||
|
|
serial query). Resolver hit its 10 s internal retry, gave up, exited
|
|||
|
|
with code 2. Powermon services had `Requires=lvx-resolve-links` so
|
|||
|
|
systemd refused to start them, and there was no retry path — stack
|
|||
|
|
stayed dead until manual intervention ~10 h later.
|
|||
|
|
|
|||
|
|
2. **Recovery.** Re-ran the resolver manually; both inverters responded
|
|||
|
|
normally on the second attempt. Started powermon services; PV
|
|||
|
|
voltages (305.2 V / 307.2 V), battery voltages (54.0 V), and all the
|
|||
|
|
usual telemetry flowed back within seconds.
|
|||
|
|
|
|||
|
|
3. **Bulletproofing the resolver path.** Three coordinated changes so a
|
|||
|
|
transient morning blip can't latch the whole stack offline:
|
|||
|
|
|
|||
|
|
a. **Resolver → always-exit-0.** Best-effort policy: write whatever
|
|||
|
|
symlinks it can resolve, log warnings for the missing ones,
|
|||
|
|
always return success. Only-relink-on-change semantics so a
|
|||
|
|
no-op timer-driven run doesn't churn the symlinks. When symlinks
|
|||
|
|
*do* move (post-power-cycle), resolver auto-restarts powermon
|
|||
|
|
so its open hidraw fds re-bind.
|
|||
|
|
|
|||
|
|
b. **`Wants=` instead of `Requires=`** on both powermon services.
|
|||
|
|
Resolver failure no longer propagates as "permanently failed"
|
|||
|
|
to dependent units. Powermon's own `Restart=always` handles the
|
|||
|
|
retry loop until `/dev/lvx6048-{1,2}` appear.
|
|||
|
|
|
|||
|
|
c. **Periodic systemd timer** for the resolver
|
|||
|
|
(`lvx-resolve-links.timer`) — `OnBootSec=2min`,
|
|||
|
|
`OnUnitActiveSec=5min`. Belt-and-suspenders against any
|
|||
|
|
hot-plug case where udev didn't fire (stuck-USB scenario like
|
|||
|
|
this morning's).
|
|||
|
|
|
|||
|
|
d. **`StartLimitIntervalSec=0`** on both powermon services — disables
|
|||
|
|
systemd's default "stop trying after 5 fails in 10 s" rate-limiter,
|
|||
|
|
so the retry loop continues indefinitely until success.
|
|||
|
|
|
|||
|
|
Worst-case data gap is now ~2 min (timer cycle + couple polls) vs.
|
|||
|
|
the previous "until a human notices and runs systemctl restart".
|
|||
|
|
|
|||
|
|
4. **FWS fault polling added to powermon.** A separate gap surfaced
|
|||
|
|
during the morning's troubleshooting: front-panel showed a "32"
|
|||
|
|
indicator, and we discovered our `powermon.yaml` was polling
|
|||
|
|
`GS / MOD / PIRI / ET` only — no `FWS`. Faults / warnings weren't
|
|||
|
|
reaching MQTT at all. Added FWS (every 30 s) to both inverter
|
|||
|
|
configs.
|
|||
|
|
|
|||
|
|
5. **Defensive FWS fault-code decoder.** Same shape as the MOD `06`/`07`
|
|||
|
|
crashes from earlier in the week — an unknown fault code would
|
|||
|
|
`KeyError` and crash powermon. Pre-populated the OPTION dict with
|
|||
|
|
placeholder labels for codes 00–99, then layered the known names on
|
|||
|
|
top. Future unknown codes show as `"Fault 32"` rather than crashing.
|
|||
|
|
Also unblocks adding the four PI18 setters (PE / PD / POPM / PF) to
|
|||
|
|
the shim later — same pattern.
|
|||
|
|
|
|||
|
|
6. **About the LCD's "32".** Direct FWS queries to both units returned
|
|||
|
|
`fault_code=No fault` while `pv_loss_warning=enabled`. The LCD was
|
|||
|
|
most likely showing the PV-loss warning code (no F/W prefix on this
|
|||
|
|
firmware's display), correlating with the cold-start window. Not a
|
|||
|
|
real fault — but worth re-confirming the LCD format next time so we
|
|||
|
|
know whether 32-with-prefix is a thing on this hardware.
|
|||
|
|
|
|||
|
|
7. **EG4 lifepower discovery configs vanished.** Independent finding:
|
|||
|
|
when checking why `lvx6048_*_fault_code` wasn't appearing in HA, I
|
|||
|
|
noticed zero retained `homeassistant/sensor/lifepower4_*/config`
|
|||
|
|
topics on the broker. HA had cached entity registrations and was
|
|||
|
|
showing live state, but the canonical discovery configs were gone —
|
|||
|
|
likely cleared by an earlier `eg4-purge-orphans` invocation or
|
|||
|
|
broker retention loss. InfluxDB exporter relies on those configs to
|
|||
|
|
know what entities to publish, so pack voltages / SoCs had stopped
|
|||
|
|
reaching InfluxDB.
|
|||
|
|
|
|||
|
|
Fix: `systemctl restart eg4-battery.service`. Daemon re-publishes
|
|||
|
|
all 207 retained discovery configs at startup (3 packs × ~69
|
|||
|
|
entities). HA refreshes its registry, InfluxDB resumes capturing.
|
|||
|
|
|
|||
|
|
## Net effect
|
|||
|
|
|
|||
|
|
| Layer | Before | After |
|
|||
|
|
|---|---|---|
|
|||
|
|
| Morning-blip recovery | manual `systemctl restart` (~10 h gap) | automatic ~2 min gap |
|
|||
|
|
| Resolver exit on missing serial | code 2 (systemd marks unit failed) | always 0 (best-effort) |
|
|||
|
|
| Resolver runs | only on udev hot-plug | + every 5 min via timer |
|
|||
|
|
| Powermon ↔ resolver coupling | `Requires=` (failure cascades) | `Wants=` (independent retry) |
|
|||
|
|
| Powermon retry budget | ~5 attempts (default StartLimit) | unbounded (`StartLimitIntervalSec=0`) |
|
|||
|
|
| Fault-code → MQTT | not polled | FWS every 30 s |
|
|||
|
|
| Unknown fault code | `KeyError` crash | shows as `"Fault NN"` |
|
|||
|
|
| EG4 discovery configs | missing (purged) | 207 retained, InfluxDB unblocked |
|
|||
|
|
|
|||
|
|
## Files touched
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
M LVX6048/bin/lvx-resolve-links (always-exit-0; relink-on-change; auto-restart powermon if symlinks move)
|
|||
|
|
M LVX6048/config/powermon/powermon.yaml (added FWS poll @ 30s)
|
|||
|
|
M LVX6048/config/powermon/powermon2.yaml (same)
|
|||
|
|
M LVX6048/etc/systemd/system/powermon.service (StartLimitIntervalSec=0)
|
|||
|
|
M LVX6048/etc/systemd/system/powermon.service.d/10-resolver.conf (Requires= -> Wants=)
|
|||
|
|
M LVX6048/etc/systemd/system/powermon2.service (same)
|
|||
|
|
M LVX6048/etc/systemd/system/powermon2.service.d/10-resolver.conf (same)
|
|||
|
|
M LVX6048/install.sh (installs the timer)
|
|||
|
|
M LVX6048/powermon-patches/pi18.py (defensive fault-code decoder, 00-99)
|
|||
|
|
A LVX6048/etc/systemd/system/lvx-resolve-links.timer (every 5 min)
|
|||
|
|
A LVX6048/2026-04-28.md (this file)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## How to verify
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# Timer running?
|
|||
|
|
systemctl list-timers lvx-resolve-links.timer
|
|||
|
|
|
|||
|
|
# Resolver idempotent? (no powermon bounce on no-change)
|
|||
|
|
sudo systemctl restart lvx-resolve-links.service
|
|||
|
|
journalctl -u lvx-resolve-links.service -n 8 --no-pager
|
|||
|
|
# Expect: "(unchanged)" lines, no "restarting powermon" line.
|
|||
|
|
|
|||
|
|
# Wants= confirmed?
|
|||
|
|
systemctl show powermon.service -p Wants -p Requires
|
|||
|
|
# Expect: Wants=...lvx-resolve-links.service... ; Requires= line should NOT
|
|||
|
|
# include lvx-resolve-links.
|
|||
|
|
|
|||
|
|
# Synthetic morning-blip test (optional, disruptive):
|
|||
|
|
sudo rm /dev/lvx6048-{1,2}
|
|||
|
|
sudo systemctl restart powermon.service
|
|||
|
|
# Expect: powermon enters Restart loop; within 5 min the timer fires the
|
|||
|
|
# resolver, recreates symlinks, powermon's next retry succeeds.
|
|||
|
|
|
|||
|
|
# Fault polling alive
|
|||
|
|
mosquitto_sub -h <broker> -u mqtt -P <pass> -W 35 -v \
|
|||
|
|
-t 'homeassistant/sensor/lvx6048_1_fault_code/state' \
|
|||
|
|
-t 'homeassistant/sensor/lvx6048_2_fault_code/state'
|
|||
|
|
# Expect: "No fault" published every 30 s on each unit.
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Loose ends / next pass
|
|||
|
|
|
|||
|
|
- **FWS warning booleans aren't getting individual HA entities.** The 16
|
|||
|
|
`enabled/disabled` flags (PV loss, over-temp, fan locked, …) are
|
|||
|
|
decoded by powermon's screen formatter but the hass formatter filters
|
|||
|
|
them because they lack `unit_of_measurement` / `device_class` /
|
|||
|
|
`reading_type` metadata. Worth a separate pass to expose them as
|
|||
|
|
`binary_sensor` entities so HA gets a real "PV loss" indicator
|
|||
|
|
alongside `fault_code`.
|
|||
|
|
- **LCD code format confirmation.** Next time you can look at the LCD,
|
|||
|
|
check whether warning codes display as bare "32", "W32", or
|
|||
|
|
something else — informs whether to add a friendly mapping for
|
|||
|
|
warning codes in the docs.
|
|||
|
|
- **Faster timer cadence?** Currently 5 min (worst-case ~2 min gap).
|
|||
|
|
Could drop to 1–2 min if morning blips become more frequent. Trade-
|
|||
|
|
off: more steady-state resolver runs (each costs one PI18 ID query
|
|||
|
|
per inverter, harmless but visible in logs).
|
|||
|
|
- **eg4-battery discovery-config retention.** The configs DID get
|
|||
|
|
purged at some point — not certain what triggered it. Next
|
|||
|
|
investigation: confirm whether `eg4-purge-orphans` is over-eager,
|
|||
|
|
whether the broker is dropping retains, or whether a daemon restart
|
|||
|
|
is needed. Worth a `mosquitto_sub` watch on the broker for retain
|
|||
|
|
changes.
|