From cd3da566263912bf40ebe82f0149b95dd7b24122 Mon Sep 17 00:00:00 2001 From: Pierre-Yves Chibon Date: Thu, 13 Feb 2020 09:34:07 +0100 Subject: [PATCH] Document how to free some space on a node in openshift Signed-off-by: Pierre-Yves Chibon --- free_space_in_openshift.md | 56 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 free_space_in_openshift.md diff --git a/free_space_in_openshift.md b/free_space_in_openshift.md new file mode 100644 index 0000000..6a1478f --- /dev/null +++ b/free_space_in_openshift.md @@ -0,0 +1,56 @@ +# How to free some space in openshift? + +If your builds run into an error looking like: +``` +error: build error: devmapper: Thin Pool has 11561 free data blocks which is +less than minimum required 11944 free data blocks. Create more free space in +thin pool or use dm.min_free_space option to change behavior +``` + +It is likely that one or several of the nodes has some disk space issues. + +## Check the disk space on the nodes: + +You can do this via ansible by simply running: +``` +ansible -a 'lvs' os_nodes_stg +``` + +The output will look somthing like this: +``` + LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert + root GuestVolGroup00 -wi-ao---- <58.59g + docker-pool vg-docker twi-a-t--- 58.32g 42.00 18.44 + +os-node02.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> + LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert + root GuestVolGroup00 -wi-ao---- <58.59g + docker-pool vg-docker twi-a-t--- <48.60g 32.37 14.81 + +os-node01.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> + LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert + root GuestVolGroup00 -wi-ao---- <58.59g + docker-pool vg-docker twi-a-t--- 58.32g 40.75 17.38 + +os-node04.stg.phx2.fedoraproject.org | CHANGED | rc=0 >> + LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert + root GuestVolGroup00 -wi-ao---- <58.59g + docker-pool vg-docker twi-a-t--- 58.32g 90.32 28.35 +``` + +In this case you can see that ``os-node04.stg`` has a disk full at 90%. + + +## Free space on a node + +There is a cron job running regularly to clean up docker images that aren't +needed anymore, but you can also run it manually to free some space. + +The command to run is then: +``` +docker rmi $(docker images --filter dangling=true -q) +``` + + + +Source: https://pagure.io/fedora-infrastructure/issue/8643