Clusterpirate

Why As Kubernetes slowly moves away from Docker, it is important to know a few ctr and crictl commands in order to troubleshoot problems. I have used Docker for over 6 years in production and don't usually need to look up commands like stats, load, save, ps etc. However, without the familiar Docker command line it can be hard to perform these tasks, especially when under pressure debugging a production issue. So I wrote the following shell script hack below. As containerd is name spaced, some commands need to include the namespace k8s.io to provide a useful response. Containerd also knows the concept of pods, so the pause containers are not visible for certain commands. #!/bin/bash ARG=$1 MSG="##### No docker installed, running this for you instead:" case $ARG in info) echo "$MSG ctr version" ctr version ;; ps) echo "$MSG ctr -n k8s.io containers list - or crictl pods" crictl pods ;; rm) echo "$MSG ctr -n k8s.io containers rm

Disclaimer I have published this post on my work blog https://reece.tech previously. Overview I have operated multiple on-premise and cloud hosted K8s clusters for many years and we heavily utilise   docker   as our container runtime for master and worker nodes. As most readers would be aware by now, the Kubernetes update to 1.20 also announced the deprecation and future removal of the much loved  docker  interface. This post documents our journey from docker to a suitable replacement option. Options The two most obvious alternatives are  cri-o  and  containerd . As  containerd  is the default for many cloud based K8s environments and  containerd  was used behind the scenes by our K8s docker layer already anyway, the choice was quite easy. Changes required The main change (for K8s 1.19.5) was to install  containerd  instead of  dockerd  and then start kubelet with additional  --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock  command line options.

Why Sometimes running commands across a range of servers is necessary. Yes, there is Ansible, Puppet and SaltStack etc., however in some cases this is overkill (and these usually require python, ruby or other languages installed. The following shell script runs commands (via sudo if required) on a group of hosts in parallel. It is quite old and not very elegant but does the trick and hopefully helps somebody in the future. Please don't comment on syntax and bugs. Configuration The .run file in your home directory contains one line for user: and one line for pass: . Should this file not exist, the script will ask for the user and password interactively. The servers file includes one line per host (or includes for additional host files) - or if the file does not exist it will be treated as single host name. Usage ./run.sh server "uptime" Script #!/bin/bash # # About: Script to concurrently run commands on a list of servers (with or without sudo) # Author: Lonestarr # Las

  Disclaimer I have published this post on my work blog https://reece.tech previously. Overview Eksctl acts as a wrapper around CloudFormation templates. Creating a cluster will add one stack for the control plane (EKS master servers) and one stack for each node group configured (a node group is a group of workers using the same networking and sizing as well as IAM permissions). However, certain actions such as upgrading the Kubernetes master or worker version or scaling out the number of workers in a node group does not always update the CF stacks associated with it. Preparation Download and install the latest version of eksctl. Follow the Weaveworks installation guide:  https://eksctl.io/introduction/installation/ . Download eksctl (Linux) curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp Install eksctl (Linux) sudo mv /tmp/eksctl /usr/local/bin Provide AWS credentials Ensure th

Disclaimer I have published this post on my work blog https://reece.tech previously. Intro We are running an on premise Kubernetes cluster on Red Hat Linux 7.5 (in VMware). The  /var/lib/docker  file-system is a separate partition, formatted with  ext4  and we used  overlay  as storage provider for docker, which was recommended for earlier RHEL 7 releases. What happened One fine day, one of our containers started creating core dumps - about 1 GB per minute worth, resulting in  /var/lib/docker  (100 GB in size) to fill up in less than 90 minutes. Existing pods crashed, new pods could not pull their image or start up. We deleted the existing pods on one of the Kubernetes worker nodes manually, however the container in question migrated to a different worker and continued its mission. Investigation We believed there is a 10 GB size limit for each running containers by default, however this did not seem to be the case. After consulting the relevant documentation it became clear that the 

Disclaimer I have published this post on my work blog https://reece.tech previously. Why is this important We are running an on premise Kubernetes cluster (currently version 1.11.6) on Red Hat Linux 7.5 (in VMware). Most documentation (especially when it comes to master version upgrades) mentions checking that the control plane is healthy prior to performing any cluster changes. Obviously this is an important step to ensure consistency and repeatability - and also important during day to day management of your cluster, but how exactly do we do this? Our approach Our (multi master) Kubernetes control plane consists of a few different services / parts like etcd, kube-apiserver, scheduler, controller-manager and so on. Each component should be verified during this process. Starting simple Run  kubectl get nodes -o wide  to ensure all nodes are  Ready . Also check that the  master  servers have the  master  role. Also running  kubectl get cs  will show you the status of vital control plan

  Disclaimer I have published this post on my work blog https://reece.tech previously. Overview I like to upgrade our Kubernetes clusters quite frequently. Recently I started the upgrade journey to 1.16. Some upgrades are rather uneventless and completed within in a few minutes (we run 5 master nodes per cluster), however this particular upgrade was different. Preparation The biggest change in 1.16 is that certain (and commonly used) API versions have been removed completely. Yes, there were mentions and deprecation warnings here and there in the past but now it’s for real. For example, you will not be able to create or upgrade deployments or daemonsets created with the  extensions/v1beta1  API version without changing your resource manifests. We did upgrade Kubernetes internal services like Grafana, Prometheus, dashboards and our logging services API versions prior to upgrading our clusters to 1.16. API version changes Here is a list of all changes (removed APIs in Kubernetes): Resou