If you've ever stared at a Linux box hanging at boot for 90 seconds waiting for "network to be configured" and wondered which interface is the holdup, congratulations -- you've joined the rest of us in systemd-networkd-wait-online purgatory.

The Problem

systemd-networkd-wait-online.service is one of those units that works perfectly until it doesn't. When it blocks, you get a cryptic timeout and zero useful information about why. On a multi-homed router with VLANs, bridges, WireGuard tunnels, and a VXLAN or two, "waiting for network" could mean literally any of a dozen interfaces.

In my case, it's almost always AT&T deciding that DHCPv4 is optional today. Yes, legacy IP -- the protocol that refuses to die and refuses to work reliably. My dual-stack routers run both AT&T and Astound (formerly RCN/Grande) uplinks, and AT&T's DHCP server has the reliability of a wet paper bag. IPv6 comes up instantly via SLAAC like a civilized protocol. But DHCPv4? That's a coin flip.

The maddening part: without debug logging, you'd never know it was specifically the AT&T interface holding everything up. You just see a timeout and get to play "guess which NIC is sad today."

• To be fair - networkctl status is generally accurate early boot if an internal NIC is up to get on to the box to debug or if you're on the console ...

The Fix: One Systemd Drop-In

The solution is embarrassingly simple. Deploy a systemd drop-in that sets SYSTEMD_LOG_LEVEL=debug for the wait-online service:

# /etc/systemd/system/systemd-networkd-wait-online.service.d/debug-logging.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

That's it. Now journalctl -u systemd-networkd-wait-online.service actually tells you what's happening -- which interfaces it's watching, which ones are ready, which ones are still waiting, and what state each one is in.

Deploying It With Ansible

Because I manage my infrastructure like a reasonable person (read: obsessively with Ansible), this is a three-task addition to the networkd role:

- name: Create systemd-networkd-wait-online.service.d directory
  file:
    path: /etc/systemd/system/systemd-networkd-wait-online.service.d
    state: directory
    owner: root
    group: root
    mode: '0755'

- name: Add SYSTEMD_LOG_LEVEL=debug to systemd-networkd-wait-online
  copy:
    content: |
      [Service]
      Environment=SYSTEMD_LOG_LEVEL=debug
    dest: /etc/systemd/system/systemd-networkd-wait-online.service.d/debug-logging.conf
    owner: root
    group: root
    mode: '0644'
  register: wait_online_debug

- name: Reload systemd daemon for wait-online drop-in
  systemd:
    daemon_reload: yes
  when: wait_online_debug.changed

This runs on every managed host. No when: guard needed -- if you're running systemd-networkd, you want this. The debug output is only emitted during the boot wait, so it's not spamming your journal 24/7.

What The Debug Output Actually Shows You

After a reboot, journalctl -b -u systemd-networkd-wait-online.service gives you a per-interface breakdown of what wait-online is monitoring and what state each link is in. It explicitly logs when an interface reaches "routable" or "degraded" state, and which ones it's still waiting on.

Example: home2.cooperlees.com Reboot Log

-- Boot 6c997bfcdab84e40b7a13eff60023617 --
Mar 27 18:58:17 home2.cooperlees.com systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Mar 27 18:58:29 home2.cooperlees.com systemd[1]: Finished systemd-networkd-wait-online.service - Wait for Network to be Configured.
Mar 27 19:05:31 home2.cooperlees.com systemd[1]: systemd-networkd-wait-online.service: Deactivated successfully.
Mar 27 19:05:31 home2.cooperlees.com systemd[1]: Stopped systemd-networkd-wait-online.service - Wait for Network to be Configured.

# Vs.

-- Boot cd3ab5938f32483189a060ebaf60e37e --
Mar 27 19:08:14 home2.cooperlees.com systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: Found link lo(1)
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: Found link enp2s0(2)
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: Found link wlp1s0(3)
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: wlp1s0: link has not yet been processed by udev: setup state is pending.
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: Found link br-k8(4)
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to determine whether the link is required for online or not, assuming required: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to get required operational state, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to get operational state, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to get required address family, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to get IPv4 address state, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to get IPv6 address state, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to get setup state, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: br-k8: Failed to update link state, ignoring: No such file or directory
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: wlp1s0: link has not yet been processed by udev: setup state is pending.
Mar 27 19:08:14 home2.cooperlees.com systemd-networkd-wait-online[1068]: Found link wg0(5)
...

• Full output: https://gist.github.com/cooperlees/c7d938a4ea5ef01ffad5f96f5027671b

The key things to look for:

• "link becomes ready" -- this interface is good to go
• "Waiting for..." -- this is your culprit
• Interface states -- routable means fully configured (has a gateway), degraded means it has an address but no route, configuring means it's still waiting for DHCP/RA

Why This Matters for Multi-Homed Setups

If you're running a single-NIC cloud VPS, you probably don't care about any of this. But if you're running a home router with:

• Dual ISP uplinks (one reliable, one AT&T)
• Multiple VLANs for network segmentation
• WireGuard tunnels to your other sites
• A bridge interface for your k8s nodes
• etc.

...then knowing exactly which interface is causing your 90-second boot delay is the difference between a quick fix and an afternoon of tcpdump.

The BOFH Take / Bullshit rant for fun

Every time AT&T's DHCP server ghosts me, I'm reminded that we've been doing DHCP since 1993 and somehow ISPs still can't keep a lease server running. Meanwhile, IPv6 SLAAC just works -- stateless, no server needed, your router sends an RA and you're done. But sure, let's keep running the entire internet on a protocol that needs a dedicated server to hand out 32-bit numbers. Legacy IP is the COBOL of networking: everyone depends on it, nobody wants to maintain it, and it breaks at the worst possible time.

The real fix is to finish the IPv6 transition so we can stop caring about DHCPv4 entirely. But until that glorious day, at least now I can see which interface is waiting for AT&T to remember how DHCP works.

TL;DR

1. Add a systemd drop-in with Environment=SYSTEMD_LOG_LEVEL=debug to systemd-networkd-wait-online.service
2. Reboot
3. Check journalctl -b -u systemd-networkd-wait-online.service
4. See exactly which interface is holding up your boot
5. Blame AT&T's DHCPv4 server
6. Sadly I need it to NAT64 ... boo ...