Skip to main content

Posts

Showing posts with the label Limit

Docker Container Size Quota within Kubernetes

Disclaimer I have published this post on my work blog https://reece.tech previously. Intro We are running an on premise Kubernetes cluster on Red Hat Linux 7.5 (in VMware). The  /var/lib/docker  file-system is a separate partition, formatted with  ext4  and we used  overlay  as storage provider for docker, which was recommended for earlier RHEL 7 releases. What happened One fine day, one of our containers started creating core dumps - about 1 GB per minute worth, resulting in  /var/lib/docker  (100 GB in size) to fill up in less than 90 minutes. Existing pods crashed, new pods could not pull their image or start up. We deleted the existing pods on one of the Kubernetes worker nodes manually, however the container in question migrated to a different worker and continued its mission. Investigation We believed there is a 10 GB size limit for each running containers by default, however this did not seem to be the case. After consulting the relevant documentation it became clear that the