How to free disk space in Kubernetes nodes

The alert

Node has a filesystem that is not read-only which is (a) above 60% usage and (b) is expected to reach 100% usage in 24h.

alert: NodeFilesystemSpaceFillingUp
expr: |
    (
        node_filesystem_avail_bytes{fstype!="",job="node-exporter"} / node_filesystem_size_bytes{fstype!="",job="node-exporter"} * 100 < 40
        and
        predict_linear(node_filesystem_avail_bytes{fstype!="",job="node-exporter"}[6h], 24 * 60 * 60) < 0
        and
        node_filesystem_readonly{fstype!="",job="node-exporter"} == 0
    )
for: 1h

The investigation

The disk is above 80% utilization:

# disk usage above 80%
[root@ip-172-23-91-119 docker]# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  100G   81G   20G  81% /

~65% of disk usage is due to Docker images: