Recently I encountered a problem with mounting NFS filesystems at boot. In my case – and that of a few people who have asked related questions at various fora – the issue had to do with boot dependencies and race conditions. In this post I am explaining my issue and how I resolved it.
I first found this issue when rebuilding my local development environment, including a few VirtualBox machines that I use for testing various things. I tend to use VMs with a Host-only Adapter as well as additional network interfaces depending on what I need to accomplish.
New VMs would throw an error like this at boot:
[FAILED] Failed to mount /home/lem/go.
See 'systemctl status home-lem-go.mount' for details.
[DEPEND] Dependency failed for Remote File Systems.
And of course, my shared Go directory would not be mounted on the VM. After boot, A simple mount -a would mount the go filesystem with no issues, which was both puzzling and annoying at first. System logs would include lines such as this:
Jun 9 02:16:42 base mount[462]: mount.nfs: Network is unreachable
Turns out that these log entries would prove to be a great clue on what was going on behind the scenes.
The relationship between systemd and /etc/fstab
The introduction of systemd has brought many changes in the way Linux systems work. This is one of those areas.
Nowadays, systemd-fstab-generator automatically parses lines in the /etc/fstab file and generates equivalent systemd.mount stanzas that allow systemd to mount filesystems in order. This generally works well for filesystems that reside in local storage, fully available by the time the mount invocation comes. Remote filesystems are a slightly different story though.
The systemctl tool is useful to let us peek in the reasons behind the specific ordering of tasks at boot. In this case, we can see what would be required for the remote-fs.target to be satisfied:
systemctl list-dependencies --after remote-fs.target
remote-fs.target
├─home-lem-go.mount
└─remote-fs-pre.target
├─rpcbind.service
└─nfs-client.target
├─gssproxy.service
├─rpc-gssd.service
└─rpc-svcgssd.service
So, in order to finish with target remote-fs.target, I would need to complete task home-lem-go.mount. And what about this one, derived from the corresponding line in /etc/fstab?
systemctl list-dependencies --after home-lem-go.mount
home-lem-go.mount
├─-.mount
├─system.slice
├─systemd-journald.socket
├─network-online.target
│ ├─networking.service
│ ├─systemd-networkd-wait-online.service
│ └─network.target
│ ├─ifupdown-pre.service
│ ├─networking.service
│ ├─systemd-networkd.service
│ ├─wpa_supplicant.service
│ └─network-pre.target
├─network.target
│ ├─ifupdown-pre.service
│ ├─networking.service
│ ├─systemd-networkd.service
│ ├─wpa_supplicant.service
│ └─network-pre.target
└─remote-fs-pre.target
├─rpcbind.service
└─nfs-client.target
├─gssproxy.service
├─rpc-gssd.service
└─rpc-svcgssd.service
It seems that systemd knows to get the network configured prior to mounting this remote filesystem. Or does it?
After going through RedHat’s Bugzilla #1027114 dating back to 2014 and poking a bit on my VMs, turns out that on the current Debian distribution, none of the network-related targets actually wait for DHCP configuration to complete when using if-up style configuration. In practical terms, this means that network-online.target is reported as complete as soon as the DHCP client is launched but not necessarily after the interface is actually usable.
By the time systemd gets to execute the required mount calls, the incomplete network configuration caused the Network is unreachable messages in my logs and the failure to mount the required filesystems.
My local solution
Some suggestions found through relevant use of Google-fu include switching to automount to have the filesystems mounted on demand, or using network-manager. I think these suggestions are sidestepping the issue.
I opted for a simpler alternative. Since the interface configuration in my VMs is quite simple, I just moved away the /etc/network/interfaces file and asked systemd to take care of my interfaces via .network files. Below is a trivial /etc/systemd/network/enp0s3.network that configures my first network interface:
[Match]
Name=enp0s3
[Network]
DHCP=ipv4
When network interface configuration is left to systemd, it knows when the DHCP configuration process is complete and forces the mounting of remote filesystems to wait for the network to be actually usable. More interesting setups are possible. Check the Debian Wiki on SystemdNetworkd for more information on how to address more complex network setups.