I generally let my server do its thing, but I run into an issue consistently when I install system updates and then reboot: Some docker containers come online, while others need to be started manually. All containers were running before the system shut down.
- My containers are managed with docker compose.
- Their compose files have
restart: always - It's not always the same containers that fail to come online
- Some of them depend on an NFS mount point being ready on the host, but not all
Host is running Ubuntu Noble
Most of these containers were migrated from my previous server, and this issue never manifested.
I wonder if anyone has ideas for what to look for?
SOLVED
The issue was that docker was starting before my NFS mount point was ready, and the containers which depended on it were crashing.
Symptoms:
journalctl -b0 -u docker showed the following log lines (-b0 means to limit logs to the most recent boot):
level=error msg="failed to start container" container=fe98f37d1bc3debb204a52eddd0c9448e8f0562aea533c5dc80d7abbbb969ea3 error="error while creating mount source path '/mnt/nas/REDACTED': mkdir /mnt/nas/REDACTED: operation not permitted"
...
level=warning msg="ShouldRestart failed, container will not be restarted" container=fe98f37d1bc3debb204a52eddd0c9448e8f0562aea533c5dc80d7abbbb969ea3 daemonShuttingDown=true error="restart canceled" execDuration=5m8.349967675s exitStatus="{0 2024-10-29 00:07:32.878574627 +0000 UTC}" hasBeenManuallyStopped=false restartCount=0
I had previously set my mount directory to be un-writable if the NFS were not ready, so this lined up with my expectations.
I couldn't remember how systemd names mount points, but the following command helped me find it:
systemctl list-units -t mount | grep /mnt/nas
It gave me mnt-nas.mount as the name of the mount unit, so then I just added it to the After= and Requires= lines in my /etc/systemd/system/docker.service file:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com/
After=network-online.target docker.socket firewalld.service containerd.service time-set.target mnt-nas.mount
Wants=network-online.target containerd.service
Requires=docker.socket mnt-nas.mount
...
Looks like you've already had some help fixing your system. I'd recommend reading the arch wiki pages on encryption. Even if you're not running arch, the wiki is a good source of up-to-date information, and contains many practical tips.
This section about setting up your
/etc/crypttabto unlock a non-root filesystem at boot might be useful. Even if you don't want to unlock at boot, it explains how crypttab works: https://wiki.archlinux.org/title/Dm-crypt/System_configuration#crypttab.There's another useful section if you want to decrypt on demand after boot: https://wiki.archlinux.org/title/Dm-crypt/System_configuration#Mounting_on_demand
Lastly, if you're experimenting with fstab entries for a non-root device, consider setting
nofailso that if you screw anything up the system still boots. This section is somewhat relevant: https://wiki.archlinux.org/title/Fstab#External_devicesBasically, you need both a crypttab entry and an fstab entry to open and then mount the device.