Recently I encountered a problem with mounting NFS filesystems at boot. In my case – and that of a few people who have asked related questions at various fora – the issue had to do with boot dependencies and race conditions. In this post I am explaining my issue and how I resolved it.
I first found this issue when rebuilding my local development environment, including a few VirtualBox machines that I use for testing various things. I tend to use VMs with a Host-only Adapter as well as additional network interfaces depending on what I need to accomplish.
New VMs would throw an error like this at boot:
[FAILED] Failed to mount /home/lem/go. See 'systemctl status home-lem-go.mount' for details. [DEPEND] Dependency failed for Remote File Systems.
And of course, my shared Go directory would not be mounted on the VM. After boot, A simple
mount -a would mount the
go filesystem with no issues, which was both puzzling and annoying at first. System logs would include lines such as this:
Jun 9 02:16:42 base mount: mount.nfs: Network is unreachable
Turns out that these log entries would prove to be a great clue on what was going on behind the scenes.
The relationship between
The introduction of
systemd has brought many changes in the way Linux systems work. This is one of those areas.
systemd-fstab-generator automatically parses lines in the
/etc/fstab file and generates equivalent
systemd.mount stanzas that allow
systemd to mount filesystems in order. This generally works well for filesystems that reside in local storage, fully available by the time the
mount invocation comes. Remote filesystems are a slightly different story though.
systemctl tool is useful to let us peek in the reasons behind the specific ordering of tasks at boot. In this case, we can see what would be required for the
remote-fs.target to be satisfied:
systemctl list-dependencies --after remote-fs.target remote-fs.target ├─home-lem-go.mount └─remote-fs-pre.target ├─rpcbind.service └─nfs-client.target ├─gssproxy.service ├─rpc-gssd.service └─rpc-svcgssd.service
So, in order to finish with target
remote-fs.target, I would need to complete task
home-lem-go.mount. And what about this one, derived from the corresponding line in
systemctl list-dependencies --after home-lem-go.mount home-lem-go.mount ├─-.mount ├─system.slice ├─systemd-journald.socket ├─network-online.target │ ├─networking.service │ ├─systemd-networkd-wait-online.service │ └─network.target │ ├─ifupdown-pre.service │ ├─networking.service │ ├─systemd-networkd.service │ ├─wpa_supplicant.service │ └─network-pre.target ├─network.target │ ├─ifupdown-pre.service │ ├─networking.service │ ├─systemd-networkd.service │ ├─wpa_supplicant.service │ └─network-pre.target └─remote-fs-pre.target ├─rpcbind.service └─nfs-client.target ├─gssproxy.service ├─rpc-gssd.service └─rpc-svcgssd.service
It seems that
systemd knows to get the network configured prior to mounting this remote filesystem. Or does it?
After going through RedHat’s Bugzilla #1027114 dating back to 2014 and poking a bit on my VMs, turns out that on the current Debian distribution, none of the network-related targets actually wait for DHCP configuration to complete when using
if-up style configuration. In practical terms, this means that
network-online.target is reported as complete as soon as the DHCP client is launched but not necessarily after the interface is actually usable.
By the time
systemd gets to execute the required
mount calls, the incomplete network configuration caused the
Network is unreachable messages in my logs and the failure to mount the required filesystems.
My local solution
Some suggestions found through relevant use of Google-fu include switching to
automount to have the filesystems mounted on demand, or using
network-manager. I think these suggestions are sidestepping the issue.
I opted for a simpler alternative. Since the interface configuration in my VMs is quite simple, I just moved away the
/etc/network/interfaces file and asked
systemd to take care of my interfaces via
.network files. Below is a trivial
/etc/systemd/network/enp0s3.network that configures my first network interface:
[Match] Name=enp0s3 [Network] DHCP=ipv4
When network interface configuration is left to
systemd, it knows when the DHCP configuration process is complete and forces the mounting of remote filesystems to wait for the network to be actually usable. More interesting setups are possible. Check the Debian Wiki on SystemdNetworkd for more information on how to address more complex network setups.