Skip to main content

Posts

Showing posts from 2021

Deprecating Networking Ingress API version in Kubernetes 1.22

  Intro Kubernetes deprecates API versions over time. Usually this affects alpha and beta versions and only requires changing the apiVersion: line in your resource file to make it work. However with this Ingress object version change, additional changes are necessary. Basics For this post I am quickly creating a new cluster via Kind (Kubernetes in Docker) . Once done, we can see which API versions are supported by this cluster (version v1.21.1). $ kubectl api-versions | grep networking networking.k8s.io/v1 networking.k8s.io/v1beta1 Kubernetes automatically converts existing resources internally into different supported API versions. So if we create a new Ingress object with version v1beta1 on a recent cluster version, you will receive a deprecation warning - and the same Ingress object will exist both in version v1beta1 and v1. Create $ cat ingress_beta.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:   name: clusterpirate-ingress spec:   rules:   - http:       path

Analysing and replaying MySQL database queries using tcpdump

Why There are situations where you want to quickly enable query logging on a MySQL Database or trouble shoot queries hitting the Database server in real-time. Yes, you can enable the DB query log and there are other options available, however the script below has helped me in many cases as it is non intrusive and does not require changing the DB server, state or configuration in any way. Limitations The following only works if the DB traffic is not encrypted (no SSL/TLS transport enabled). Also this needs to be run directly on the DB server host (as root / admin). Please also be aware that this should be done on servers and data you own only. Script This script has been amended to suit my individual requirements. #!/bin/sh tcpdump -i any -s 0 -l -w - dst port 3306 | strings | perl -e ' while(<>) { chomp; next if /^[^ ]+[ ]*$/;   if(/^(ALTER|COMMIT|CREATE|DELETE|DROP|INSERT|SELECT|SET|UPDATE|ROLLBACK)/i) {     if (defined $q) { print "$q\n"; }     $q=$_;   } else {  

Exporting and importing docker images manually

  Why Sometimes it can be handy to have a copy of a container image locally or being able to manually copy a docker image from one computer to another.  Recently I had an issue where newly built Kubernetes worker nodes did not work properly due to the fact that the flannel pod image was hosted on quay.io , which was not available at the time. The "fix" was to manually export the image from a server which had flannel running just fine and import on the new worker nodes (and restart the flannel pods). Export Assuming we want to save / export the image below: $ docker images REPOSITORY     TAG       IMAGE ID       CREATED       SIZE kindest/node   <none>    af39c553b6de   2 weeks ago   1.12GB We run docker save with the image id and redirect the output into a new local file. $ docker save af39c553b6de > kindest-node.tar Once done, we end up with a new tar file, which can be compressed optionally. $ ls -lah kindest-node.tar  -rw-rw-r-- 1 user user 1.1G Jun  6 12:11 kinde

Create a Kubernetes cluster using kind (Kubernetes in Docker) in less than 2 minutes

Why Sometimes I just need to quickly test a K8s resource or compare a cluster with a near vanilla version. This is where kind comes in handy, as it can create a clean and fresh Kubernetes cluster in under 2 minutes. Requirements You have a working docker environment. Step 1 Download the kind binary (less than 4 MB). curl -Lso ./kind https://kind.sigs.k8s.io/dl/v0.11.0/kind-linux-amd64 && chmod 755 kind Step 2 Create the actual cluster. $ time ./kind create cluster Creating cluster "kind" ...  ✓ Ensuring node image (kindest/node:v1.21.1)  ✓ Preparing nodes  ✓ Writing configuration  ✓ Starting control-plane  ✓ Installing CNI  ✓ Installing StorageClass Set kubectl context to "kind-kind" You can now use your cluster with: kubectl cluster-info --context kind-kind Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community real     1m55.934s user    0m1.014s sys     0m0.970s Step 3 That's it really - just use  kubectl   (ideally

Building Kubernetes Clusters using Kubespray

Disclaimer I have published this post on my work blog https://reece.tech previously. Preface We are hosting our workloads in Docker containers within various Kubernetes clusters. To ensure consistency and repeatability across environments, we are using idempotent configuration management tools like Ansible. Kubespray is an Ansible playbook used to manage Kubernetes clusters including initial build and also lifecyle of the cluster (adding or removing nodes, version upgrades etc.). Requirements We are automatically provisioning new VMs in our VSphere environment using Ansible. Once the new node is up and running, Kubespray runs across to install required Kubernetes services. In this example we are using a root user ssh key for passwordless authentication. Ansible 2.5 Kubespray git repository Steps Getting Kubespray The following commands will download Kubespray, select the latest release version and install necessary Python modules. git clone https://github.com/kubernetes-sigs/kubespray

Docker wrapper for containerd and common commands

Why As Kubernetes slowly moves away from Docker, it is important to know a few ctr and crictl commands in order to troubleshoot problems. I have used Docker for over 6 years in production and don't usually need to look up commands like stats, load, save, ps etc. However, without the familiar Docker command line it can be hard to perform these tasks, especially when under pressure debugging a production issue. So I wrote the following shell script hack below. As containerd is name spaced, some commands need to include the namespace k8s.io to provide a useful response. Containerd also knows the concept of pods, so the pause containers are not visible for certain commands. #!/bin/bash ARG=$1 MSG="##### No docker installed, running this for you instead:" case $ARG in info) echo "$MSG ctr version" ctr version ;; ps) echo "$MSG ctr -n k8s.io containers list - or crictl pods" crictl pods ;; rm) echo "$MSG ctr -n k8s.io containers rm

Migrating Kubernetes from Docker to Containerd

Disclaimer I have published this post on my work blog https://reece.tech previously. Overview I have operated multiple on-premise and cloud hosted K8s clusters for many years and we heavily utilise   docker   as our container runtime for master and worker nodes. As most readers would be aware by now, the Kubernetes update to 1.20 also announced the deprecation and future removal of the much loved  docker  interface. This post documents our journey from docker to a suitable replacement option. Options The two most obvious alternatives are  cri-o  and  containerd . As  containerd  is the default for many cloud based K8s environments and  containerd  was used behind the scenes by our K8s docker layer already anyway, the choice was quite easy. Changes required The main change (for K8s 1.19.5) was to install  containerd  instead of  dockerd  and then start kubelet with additional  --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock  command line options.

Quick and simple parallel ssh command shell script

Why Sometimes running commands across a range of servers is necessary. Yes, there is Ansible, Puppet and SaltStack etc., however in some cases this is overkill (and these usually require python, ruby or other languages installed. The following shell script runs commands (via sudo if required) on a group of hosts in parallel. It is quite old and not very elegant but does the trick and hopefully helps somebody in the future. Please don't comment on syntax and bugs. Configuration The .run file in your home directory contains one line for user: and one line for pass: . Should this file not exist, the script will ask for the user and password interactively. The servers file includes one line per host (or includes for additional host files) - or if the file does not exist it will be treated as single host name. Usage ./run.sh server "uptime" Script #!/bin/bash # # About: Script to concurrently run commands on a list of servers (with or without sudo) # Author: Lonestarr # Las

Building EKS (Amazon hosted Kubernetes) clusters using eksctl

  Disclaimer I have published this post on my work blog https://reece.tech previously. Overview Eksctl acts as a wrapper around CloudFormation templates. Creating a cluster will add one stack for the control plane (EKS master servers) and one stack for each node group configured (a node group is a group of workers using the same networking and sizing as well as IAM permissions). However, certain actions such as upgrading the Kubernetes master or worker version or scaling out the number of workers in a node group does not always update the CF stacks associated with it. Preparation Download and install the latest version of eksctl. Follow the Weaveworks installation guide:  https://eksctl.io/introduction/installation/ . Download eksctl (Linux) curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp Install eksctl (Linux) sudo mv /tmp/eksctl /usr/local/bin Provide AWS credentials Ensure th