coreos-ci openshift namespace PVC is 85% full

[dustymabe]

From Kevin Fenzi in matrix:

FYI, the coreos-ci namespace has a pvc called 'jenkins' that is now 85% full. Let me know if you would like me to grow it or if you want to deal with it.

Can someone look into this?

[marmijo]

I cleaned up space in the coreos-ci jenkins PVE. Here's where it stands now:

/dev/rbd3        25G   13G   12G  54% /var/lib/jenkins

Since the 'Delete' options are currently restricted/disabled in our Jenkins UI, I had to clear the storage manually via the Pod terminal. I targeted old builds in /var/lib/jenkins/jobs/${REPO}/branches/PR-XXXX directories and deleted only builds that were at least 14 days old.

cd /var/lib/jenkins/jobs/${REPO}/branches/PR-XXXX

# FILES_THAT_ARE_NOT_BUILDS = files in the ${REPO}/branches directory that are not builds (`legacyIds`, `permalinks`). 
# I assume these are still needed if the PR is open...

find . -maxdepth 1 -mtime +13 | grep -v ${FILES_THAT_ARE_NOT_BUILDS}

# I verified each build/job was old and no longer in-use before running the delete command. (repeat the last command: https://www.redhat.com/en/blog/bash-bang-commands)
rm -rf $(!!)

Afterwards, I cycled the jenkins pod to "reload the configuration from disk" so the builds didn't show up anymore.

[dustymabe]

Thannks! @marmijo
I'll close this out

[marmijo]

Quick follow up, I realized the primary culprit was coreos/coreos-assembler#4375. There have been a lot of commits force-pushed to this PR since January causing 157 different builds in the jenkins instance. Each of these builds is roughly 45 MB, so this one PR accounted for 27% of the total PVC usage at 6.8GB!

[marmijo]

I opened coreos/coreos-assembler#4498 to try to cap the PVC usage from the coreos-assembler repo. We could add this to the other repos that are tested in coreos-ci if needed.