castris
Verified User
Summary
When
Result: additional IPs declared in
This is not a legacy-release issue. It is fully reproducible on the current LTS, and the shipped unit still lacks the one directive that would prevent it.
Affected: current LTS, not just the old one
Ubuntu 24.04 LTS noble (primary target of this report) — DA 1.699+, kernel 6.8.0-111,
The package's own unit still ships without
The changelog of
Ubuntu 22.04 LTS jammy (also affected) — DA 1.699 / 1.698, kernel 5.15.0-176,
Deployment evidence
The drop-in below is currently deployed and verified across 16 DirectAdmin servers (jammy and noble), validated per host with
How to reproduce (5 min)
Pre-requisites: stock DA box on Ubuntu 24.04 (or 22.04) with at least 1 additional IP listed in
Real-world trigger frequency
Any
Upstream bug
This is a known upstream limitation of
The dispatcher subscribes to D-Bus signals from
Proposed fix (validated)
A 4-line systemd drop-in for
Then
How it works
Validation
End-to-end recovery: under 5 seconds. The drop-in is backwards compatible (no-op on systems without dispatcher).
Suggestion for DA packaging
The DA package that already installs
If you'd prefer a different approach (e.g., wrapping
Happy to provide more logs / test on additional Ubuntu versions if useful.
When
systemd-networkd is restarted on a DA install on Ubuntu 24.04 LTS (typical trigger: apt upgrade of netplan.io, libnetplan0 or systemd), networkd-dispatcher.service keeps running but silently loses its D-Bus subscription to networkd's state changes. The subsequent routable transition is never processed, so the hooks under /etc/networkd-dispatcher/routable.d/ are not invoked — including startips-networkd shipped by DA.Result: additional IPs declared in
/usr/local/directadmin/data/admin/ip.list (and managed by DA via /usr/local/directadmin/scripts/startips) get wiped from the interface and never restored until the next reboot. Customer domains using those IPs become unreachable. Let's Encrypt auto-renewals fail with "No domains pointing to this server".This is not a legacy-release issue. It is fully reproducible on the current LTS, and the shipped unit still lacks the one directive that would prevent it.
Affected: current LTS, not just the old one
Ubuntu 24.04 LTS noble (primary target of this report) — DA 1.699+, kernel 6.8.0-111,
networkd-dispatcher 2.2.4-1, systemd 255 (255.4-1ubuntu8.15).The package's own unit still ships without
PartOf=systemd-networkd.service. One command, on any stock noble box, no drop-in installed:
Bash:
grep -c PartOf /lib/systemd/system/networkd-dispatcher.service
# → 0The changelog of
networkd-dispatcher 2.2.4-1 mentions no fix related to D-Bus subscription loss on networkd restart. So on noble — supported until 2029 (ESM 2036) — the failure mode is live today, on a stock install, with nothing to prevent it.Ubuntu 22.04 LTS jammy (also affected) — DA 1.699 / 1.698, kernel 5.15.0-176,
networkd-dispatcher 2.1-2, systemd 249.11-0ubuntu3.20, netplan.io 0.107.1-3ubuntu0.22.04.3. Same behaviour. The drop-in proposed below survives a do-release-upgrade from jammy to noble unchanged and keeps working as expected.Deployment evidence
The drop-in below is currently deployed and verified across 16 DirectAdmin servers (jammy and noble), validated per host with
systemctl restart systemd-networkd → all additional IPs present at T+5s. No regressions, no side effects, across several weeks of unattended-upgrades cycles.How to reproduce (5 min)
Pre-requisites: stock DA box on Ubuntu 24.04 (or 22.04) with at least 1 additional IP listed in
ip.list and currently active on the primary interface.
Bash:
# 0) Confirm the unit ships without PartOf (this is the root cause)
grep -c PartOf /lib/systemd/system/networkd-dispatcher.service
# Result: 0
# 1) Confirm initial state
ip -4 addr show ens3 | grep "inet "
# Expected: primary IP + additional IPs all present
# 2) Trigger the bug (this is exactly what apt upgrade does after touching netplan/systemd)
systemctl restart systemd-networkd
sleep 5
# 3) Observe the failure
ip -4 addr show ens3 | grep "inet "
# Result: ONLY the primary IP remains. Additional IPs are gone.
# 4) Confirm dispatcher silently failed to process the transition
journalctl -u networkd-dispatcher --since "30 seconds ago"
# Result: NO log entries. The dispatcher process is alive but blind.
# 5) Confirm the hook is in place but never executed
ls /etc/networkd-dispatcher/routable.d/startips-networkd
# Result: file exists, executable. It just never gets called.Real-world trigger frequency
Any
apt upgrade (manual or via unattended-upgrades) that updates packages whose postinstall hooks restart systemd-networkd. Most common offenders: netplan.io, libnetplan0, systemd, systemd-sysv. On a fleet of DA boxes running unattended-upgrades, this fires every few weeks silently — until somebody notices LE is failing or a customer reports their site is down.Upstream bug
This is a known upstream limitation of
networkd-dispatcher:- https://bugs.launchpad.net/ubuntu/+source/networkd-dispatcher/+bug/1898343
- https://github.com/craftyguy/networkd-dispatcher/issues/27
The dispatcher subscribes to D-Bus signals from
systemd-networkd once at startup and doesn't recover on networkd restart.Proposed fix (validated)
A 4-line systemd drop-in for
networkd-dispatcher.service:
INI:
# /etc/systemd/system/networkd-dispatcher.service.d/follow-networkd.conf
[Unit]
PartOf=systemd-networkd.service
After=systemd-networkd.service
[Service]
Restart=on-failureThen
systemctl daemon-reload.How it works
PartOf=systemd-networkd.service makes systemd restart the dispatcher whenever systemd-networkd is restarted. After the restart, the dispatcher re-subscribes to D-Bus, picks up the current routable state (which is already there because networkd has finished bringing up the interface), and invokes the routable.d/ hooks normally — including startips-networkd, which restores the additional IPs.Validation
Code:
T0 systemctl restart systemd-networkd
T0+0s networkd-dispatcher: Stopped (PartOf triggered the cascade)
T0+0s systemd-networkd: ens3 Link UP / Gained carrier / IPv6LL / DHCP
T0+0s networkd-dispatcher: Starting again
T0+1s networkd-dispatcher: Started, processes the routable state
T0+1s /etc/networkd-dispatcher/routable.d/startips-networkd executes
T0+1s /usr/local/directadmin/scripts/startips runs
T0+5s ip -4 addr show ens3 → ALL IPs present (primary + additional)End-to-end recovery: under 5 seconds. The drop-in is backwards compatible (no-op on systems without dispatcher).
Suggestion for DA packaging
The DA package that already installs
/etc/networkd-dispatcher/routable.d/startips-networkd could install this drop-in alongside it. DA ships the hook, so DA is the natural place to ship the one directive that makes the hook actually fire. That would make every DA install on noble — and jammy, and any future LTS that still ships networkd-dispatcher without PartOf= — resilient to apt upgrade shaking networkd, with zero operator intervention.If you'd prefer a different approach (e.g., wrapping
startips as ExecStartPost= of systemd-networkd.service), I'm happy to discuss trade-offs. The drop-in approach is the smallest, most localized change that uses native systemd semantics.Happy to provide more logs / test on additional Ubuntu versions if useful.