From ed554f2a391ef308c997fcac2132feea6ab5d71d Mon Sep 17 00:00:00 2001 From: Stefan Agner Date: Wed, 31 Aug 2022 23:04:23 +0200 Subject: [PATCH] Check free disk space before starting Docker (#2097) It seems that Docker can fail to start if there is no space left on the device. Try to free up some space in that case by asking journald to limit its size to 256MiB. This should work for any storage larger than ~2.5GiB (as the journals maximum size is 10% of the disk size). It still should leave enough logs to diagnose problems if necessary. Note: We could also limit the size of the journal in first place, but that isn't sustainable: Once that space is used up, we run into the same problem again. By only asking journalctl to free up if necessary, we kinda (miss)use the journal as way to "reserve" some space which we can free up at boot if necessary. --- .../etc/systemd/system/docker.service.d/failure.conf | 1 + .../rootfs-overlay/usr/libexec/docker-disk-check | 12 ++++++++++++ 2 files changed, 13 insertions(+) create mode 100755 buildroot-external/rootfs-overlay/usr/libexec/docker-disk-check diff --git a/buildroot-external/rootfs-overlay/etc/systemd/system/docker.service.d/failure.conf b/buildroot-external/rootfs-overlay/etc/systemd/system/docker.service.d/failure.conf index 8aae38311..6baba1bb5 100644 --- a/buildroot-external/rootfs-overlay/etc/systemd/system/docker.service.d/failure.conf +++ b/buildroot-external/rootfs-overlay/etc/systemd/system/docker.service.d/failure.conf @@ -2,4 +2,5 @@ FailureAction=reboot [Service] +ExecStartPre=/usr/libexec/docker-disk-check ExecStopPost=/usr/libexec/docker-failure diff --git a/buildroot-external/rootfs-overlay/usr/libexec/docker-disk-check b/buildroot-external/rootfs-overlay/usr/libexec/docker-disk-check new file mode 100755 index 000000000..8fc8e633f --- /dev/null +++ b/buildroot-external/rootfs-overlay/usr/libexec/docker-disk-check @@ -0,0 +1,12 @@ +#!/bin/sh + +# Check if less than 128MiB is available on /mnt/data (4k block size). +if [ "$(stat -f /mnt/data -c '%f')" -lt 32768 ]; then + echo "The system is very low on disk space!" + echo "This can cause Docker to fail to start, causing a boot loop." + echo "Asking systemd-journald to free up some space." + # systemd-journald defaults to 10% of disk size. So this should free up + # space for any system which has been running for a while (presumably it + # has when there is no space) and has a disk size larger than ~2.5GiB. + journalctl --vacuum-size=256M +fi