Skip to main content

Building Kubernetes Clusters using Kubespray


Disclaimer

I have published this post on my work blog https://reece.tech previously.

Preface

We are hosting our workloads in Docker containers within various Kubernetes clusters. To ensure consistency and repeatability across environments, we are using idempotent configuration management tools like Ansible. Kubespray is an Ansible playbook used to manage Kubernetes clusters including initial build and also lifecyle of the cluster (adding or removing nodes, version upgrades etc.).

Requirements

We are automatically provisioning new VMs in our VSphere environment using Ansible. Once the new node is up and running, Kubespray runs across to install required Kubernetes services. In this example we are using a root user ssh key for passwordless authentication.

  • Ansible 2.5
  • Kubespray git repository

Steps

Getting Kubespray

The following commands will download Kubespray, select the latest release version and install necessary Python modules.

git clone https://github.com/kubernetes-sigs/kubespray.git
cd kubespray
git branch -a | grep release
git checkout release-2.8
sudo pip install -r requirements.txt

Minimum configuration

We now copy the sample inventory directory to our cluster specific inventory.

cp -rfp inventory/sample inventory/onecluster

The inventory builder script automatically updates the hosts.ini file with the IP addresses of our new cluster nodes. There should be at least three addresses supplied to build a resilient (multi master) cluster.

declare -a IPS=(10.99.1.3 10.99.1.4 10.99.1.5)
CONFIG_FILE=inventory/onecluster/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]}

echo "docker_version: latest" >> inventory/onecluster/group_vars/all/all.yml

Build

The following Ansible command should build a working cluster with three shared master / worker nodes (also each running etcd in stacked control plane configuration).

ansible-playbook -i inventory/onecluster/hosts.ini --user=root cluster.yml

Additional configuration

Host names

The default auto-generated hosts file will just call each server node1, node2 etc. This can be adjusted by replacing each name in the hosts.ini.

Kubernetes versions

If you are after a specific Kubernetes version - for example 1.11.7 (set as kube_version in group_vars/k8s-cluster/k8s-cluster.yml), Kubespray may error due to missing checksums. This can be fixed by adding sha256 checksums to the roles/download/defaults/main.yml.

roles/download/defaults/main.yml
@@ -76,6 +76,7 @@ hyperkube_checksums:
v1.12.0: f80336201f3152a5307c01f8a7206847398dde15c69b3d20c76a7d9520b60daf
+ v1.11.7: 9f4c22bc5fa6d3a70f748c652f21f702b0256b8808c481e2c06af2104d42bd36
v1.11.3: dac8da16dd6688e52b5dc510f5dd0a20b54350d52fb27ceba2f018ba2c8be692
@@ -96,6 +97,7 @@ kubeadm_checksums:
v1.12.0: 463fb058b7fa2591fb01f29f2451b054f6cbaa0f8a20394b4a4eb5d68473176f
+ v1.11.7: 37ec1273ec4ca85ba704740c62bea70f171f9b574ff1be06c65f13b384185c51
v1.11.3: 422a7a32ed9a7b1eaa2a4f9d121674dfbe80eb41e206092c13017d097f75aaec

Metrics server

group_vars/k8s-cluster/addons.yml
15c15
< metrics_server_enabled: false
---
> metrics_server_enabled: true

Helm / Tiller

group_vars/k8s-cluster/addons.yml
6c6
< helm_enabled: false
---
> helm_enabled: true

Flannel instead of Calico

group_vars/k8s-cluster/k8s-cluster.yml
74c76
< kube_network_plugin: calico
---
> kube_network_plugin: flannel

Flannel backend type vxlan

group_vars/k8s-cluster/k8s-net-flannel.yml
< # flannel_backend_type: "vxlan"
---
> flannel_backend_type: "vxlan"

Updating IP address ranges for services and pods

group_vars/k8s-cluster/k8s-cluster.yml
77c79
< kube_service_addresses: 10.233.0.0/18
---
> kube_service_addresses: 10.196.0.0/16
82c84
< kube_pods_subnet: 10.233.64.0/18
---
> kube_pods_subnet: 10.144.0.0/16

Other commands

build:
ansible-playbook -i inventory/onecluster/hosts.ini --user=root cluster.yml

scale (add nodes to hosts.ini, then):
ansible-playbook -i inventory/onecluster/hosts.ini --user=root scale.yml

removing a node:
ansible-playbook -i inventory/onecluster/hosts.ini --user=root remove-node.yml -e "node=xxxx,yyyy,zzzz"

clean / destroy cluster (careful):
ansible-playbook -i inventory/onecluster/hosts.ini --user=root reset.yml

upgrade (from 1.11.3 to 1.11.7 in this case):
ansible-playbook -i inventory/onecluster/hosts.ini --user=root upgrade-cluster.yml -e kube_version=v1.11.7
Labels:

Comments

Post a Comment

Popular posts from this blog

Manual Kubernetes TLS certificate renewal procedure

Intro Kubernetes utilizes TLS certificates to secure different levels of internal and external cluster communication.  This includes internal services like the apiserver, kubelet, scheduler and controller-manager etc. These TLS certificates are created during the initial cluster installation and are usually valid for 12 months. The cluster internal certificate authority (CA) certificate is valid for ten years. There are options available to automate certificate renewals, but they are not always utilised and these certs can become out of date. Updating certain certificates may require restarts of K8s components, which may not be fully automated either. If any of these certificates is outdated or expired, it will stop parts or all of your cluster from functioning correctly. Obviously this scenario should be avoided - especially in production environments. This blog entry focuses on manual renewals / re-creation of Kubernetes certificates. For example, the api-server certificate below...
Post a Comment
Read more

Analysing and replaying MySQL database queries using tcpdump

Why There are situations where you want to quickly enable query logging on a MySQL Database or trouble shoot queries hitting the Database server in real-time. Yes, you can enable the DB query log and there are other options available, however the script below has helped me in many cases as it is non intrusive and does not require changing the DB server, state or configuration in any way. Limitations The following only works if the DB traffic is not encrypted (no SSL/TLS transport enabled). Also this needs to be run directly on the DB server host (as root / admin). Please also be aware that this should be done on servers and data you own only. Script This script has been amended to suit my individual requirements. #!/bin/sh tcpdump -i any -s 0 -l -w - dst port 3306 | strings | perl -e ' while(<>) { chomp; next if /^[^ ]+[ ]*$/;   if(/^(ALTER|COMMIT|CREATE|DELETE|DROP|INSERT|SELECT|SET|UPDATE|ROLLBACK)/i) {     if (defined $q) { print "$q\n"; }     $q=$_; ...
Post a Comment
Read more

Deprecating Networking Ingress API version in Kubernetes 1.22

  Intro Kubernetes deprecates API versions over time. Usually this affects alpha and beta versions and only requires changing the apiVersion: line in your resource file to make it work. However with this Ingress object version change, additional changes are necessary. Basics For this post I am quickly creating a new cluster via Kind (Kubernetes in Docker) . Once done, we can see which API versions are supported by this cluster (version v1.21.1). $ kubectl api-versions | grep networking networking.k8s.io/v1 networking.k8s.io/v1beta1 Kubernetes automatically converts existing resources internally into different supported API versions. So if we create a new Ingress object with version v1beta1 on a recent cluster version, you will receive a deprecation warning - and the same Ingress object will exist both in version v1beta1 and v1. Create $ cat ingress_beta.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:   name: clusterpirate-ingress spec:   rules:  ...
Post a Comment
Read more