castris
Verified User
Summary
When
Result: additional IPs declared in
Tested on
How to reproduce (5 min)
Pre-requisites: DA box on Ubuntu 22.04+ with at least 1 additional IP listed in
Real-world trigger frequency
Any
Upstream bug
This is a known upstream limitation of
The dispatcher subscribes to D-Bus signals from
Proposed fix (validated)
A 4-line systemd drop-in for
Then
How it works
Validation
End-to-end recovery: under 5 seconds. The drop-in is backwards compatible (no-op on systems without dispatcher).
Suggestion for DA packaging
The DA package that already installs
If you'd prefer a different approach (e.g., wrapping
Happy to provide more logs / test on additional Ubuntu versions if useful.
When
systemd-networkd is restarted on a DA install on Ubuntu 22.04+ (typical trigger: apt upgrade of netplan.io, libnetplan0 or systemd), networkd-dispatcher.service keeps running but silently loses its D-Bus subscription to networkd's state changes. The subsequent routable transition is never processed, so the hooks under /etc/networkd-dispatcher/routable.d/ are not invoked — including startips-networkd shipped by DA.Result: additional IPs declared in
/usr/local/directadmin/data/admin/ip.list (and managed by DA via /usr/local/directadmin/scripts/startips) get wiped from the interface and never restored until the next reboot. Customer domains using those IPs become unreachable. Let's Encrypt auto-renewals fail with "No domains pointing to this server".Tested on
- Ubuntu 22.04.5 LTS jammy: DA 1.699 / 1.698, kernel 5.15.0-176, networkd-dispatcher 2.1-2, systemd 249.11-0ubuntu3.20, netplan.io 0.107.1-3ubuntu0.22.04.3.
- Ubuntu 24.04.4 LTS noble: DA 1.699, kernel 6.8.0-111, networkd-dispatcher 2.2.4-1, systemd 255 (255.4-1ubuntu8.15). Reproducer behaves identically on noble — the upstream service unit ships without
PartOf=systemd-networkd.service, and the changelog ofnetworkd-dispatcher 2.2.4-1does not mention any fix related to D-Bus subscription loss on networkd restart. The drop-in proposed below survives ado-release-upgradefrom jammy to noble unchanged and keeps working as expected. - Reproduced on multiple servers in our fleet over the past weeks (jammy and post-noble).
How to reproduce (5 min)
Pre-requisites: DA box on Ubuntu 22.04+ with at least 1 additional IP listed in
ip.list and currently active on the primary interface.
Bash:
# 1) Confirm initial state
ip -4 addr show ens3 | grep "inet "
# Expected: primary IP + additional IPs all present
# 2) Trigger the bug (this is exactly what apt upgrade does after touching netplan/systemd)
systemctl restart systemd-networkd
sleep 5
# 3) Observe the failure
ip -4 addr show ens3 | grep "inet "
# Result: ONLY the primary IP remains. Additional IPs are gone.
# 4) Confirm dispatcher silently failed to process the transition
journalctl -u networkd-dispatcher --since "30 seconds ago"
# Result: NO log entries. The dispatcher process is alive but blind.
# 5) Confirm the hook is in place but never executed
ls /etc/networkd-dispatcher/routable.d/startips-networkd
# Result: file exists, executable. It just never gets called.Real-world trigger frequency
Any
apt upgrade (manual or via unattended-upgrades) that updates packages whose postinstall hooks restart systemd-networkd. Most common offenders: netplan.io, libnetplan0, systemd, systemd-sysv. On a fleet of DA boxes running unattended-upgrades, this fires every few weeks silently — until somebody notices LE is failing or a customer reports their site is down.Upstream bug
This is a known upstream limitation of
networkd-dispatcher:- https://bugs.launchpad.net/ubuntu/+source/networkd-dispatcher/+bug/1898343
- https://github.com/craftyguy/networkd-dispatcher/issues/27
The dispatcher subscribes to D-Bus signals from
systemd-networkd once at startup and doesn't recover on networkd restart.Proposed fix (validated)
A 4-line systemd drop-in for
networkd-dispatcher.service:
INI:
# /etc/systemd/system/networkd-dispatcher.service.d/follow-networkd.conf
[Unit]
PartOf=systemd-networkd.service
After=systemd-networkd.service
[Service]
Restart=on-failureThen
systemctl daemon-reload.How it works
PartOf=systemd-networkd.service makes systemd restart the dispatcher whenever systemd-networkd is restarted. After the restart, the dispatcher re-subscribes to D-Bus, picks up the current routable state (which is already there because networkd has finished bringing up the interface), and invokes the routable.d/ hooks normally — including startips-networkd, which restores the additional IPs.Validation
Code:
T0 systemctl restart systemd-networkd
T0+0s networkd-dispatcher: Stopped (PartOf triggered the cascade)
T0+0s systemd-networkd: ens3 Link UP / Gained carrier / IPv6LL / DHCP
T0+0s networkd-dispatcher: Starting again
T0+1s networkd-dispatcher: Started, processes the routable state
T0+1s /etc/networkd-dispatcher/routable.d/startips-networkd executes
T0+1s /usr/local/directadmin/scripts/startips runs
T0+5s ip -4 addr show ens3 → ALL IPs present (primary + additional)End-to-end recovery: under 5 seconds. The drop-in is backwards compatible (no-op on systems without dispatcher).
Suggestion for DA packaging
The DA package that already installs
/etc/networkd-dispatcher/routable.d/startips-networkd could install this drop-in alongside it. That would make every DA install on Ubuntu 22.04+ (jammy, noble, and any future LTS that still ships networkd-dispatcher without PartOf=) resilient to apt upgrade shaking networkd, with zero operator intervention.If you'd prefer a different approach (e.g., wrapping
startips as ExecStartPost= of systemd-networkd.service), I'm happy to discuss trade-offs. The drop-in approach is the smallest, most localized change that uses native systemd semantics.Happy to provide more logs / test on additional Ubuntu versions if useful.