Skip to main content

Upgrading Kubernetes to 1.16 and decommissioned API versions

 




Disclaimer

I have published this post on my work blog https://reece.tech previously.

Overview

I like to upgrade our Kubernetes clusters quite frequently. Recently I started the upgrade journey to 1.16.

Some upgrades are rather uneventless and completed within in a few minutes (we run 5 master nodes per cluster), however this particular upgrade was different.

Preparation

The biggest change in 1.16 is that certain (and commonly used) API versions have been removed completely. Yes, there were mentions and deprecation warnings here and there in the past but now it’s for real.

For example, you will not be able to create or upgrade deployments or daemonsets created with the extensions/v1beta1 API version without changing your resource manifests.

We did upgrade Kubernetes internal services like Grafana, Prometheus, dashboards and our logging services API versions prior to upgrading our clusters to 1.16.

API version changes

Here is a list of all changes (removed APIs in Kubernetes):

ResourceAPI oldAPI newK8S version
Deploymentextensions/v1beta1 / apps/v1beta2apps/v11.16
DaemonSetextensions/v1beta1 / apps/v1beta2apps/v11.16
Ingressextensions/v1beta1networking.k8s.io/v1beta11.20
NetworkPolicyextensions/v1beta1networking.k8s.io/v11.16
PodSecurityPolicyextensions/v1beta1policy/v1beta11.16
StatefulSetextensions/v1beta1 / apps/v1beta2apps/v11.16
ReplicaSetextensions/v1beta1 / apps/v1beta2apps/v11.16

How do we find resources that need to be updated?

Resources currently available in the cluster have been applied to both the old and the new API version - so upgrading to 1.16 will not break any existing configuration.

You can explicitly query the API server using kubectl to get resources specific to a certain API version - for example kubectl get deployment.extensions –all-namespaces will show you all deployments with the extensions/* API version and kubectl get deployment.apps –all-namespaces will show you the apps/v1 deployments. The output of both may actually differ.

Just running kubectl get deployment –all-namespaces may show the extensions/* version only - especially confusing when applying a new resource with the latest API version and then checking the API version using kubectl get x -o yaml only to get returned the deprecated API version for the same resource.

What exactly needs to be changed?

Most changes are quite straight forward and only require the apiVersion: field to be updated - for example for kind: Ingress you only change apiVersion: extensions/v1beta1 to apiVersion: networking.k8s.io/v1beta1 and you’re done.

However, other resources (specifically DaemonSet and Deployment) may require additional fields - for example spec.selector. You can use kubectl convert to migrate existing templates to the new API version. However, if you are using Helm charts, this is not overly useful and templates need to be updated manually.

Third party charts

Certain third party charts (such as Ingress controllers, Prometheus and friends, operators to name a few) needed to be updated as well.

Ongoing

We are in the (slow) process of upgrading hundreds of Ingress API versions in various git repositories over the next few months to be ready for version 1.20 (when the old/deprecated Ingress API version gets removed).

Other Pitfalls

Other than the API version change, we also experienced problems with Flannel and Grafana.

Flannel

The network overlay Flannel broke immediately after the API server upgrade with error message “runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized” (and all master nodes were marked as NotReady). Not ideal.

Cause was the flannel configuration (file and config map): /etc/cni/net.d/10-flannel.conflist, which did not include the cniVersion key. Updating this (and adding “cniVersion”: “0.2.0”) fixed the issue.

Prometheus / Grafana dashboards

The upgrade also broke some of our Grafana dashboards, as the container_cpu_usage_seconds_total key removed the container_name entity (replaced by container) - so quite an easy fix.

RBAC

Some permissions may need to be created / updated for the new API versions. Our tiller permissions did not allow networking.k8s.io/v1beta1 resources to be updated for example.

Pod restarts

After restarting the new kubelet, all pods on the node in question restarted simultaneously, leading to very high node load. We decided to sequentially rebuild each node with 1.16 instead of upgrading worker nodes in place over a period of a couple of weeks instead of a big bang approach for our production clusters.

Prometheus helm charts

We had older helm charts for Prometheus installed using deprecated API versions. A few weeks after the 1.16 change, the Prometheus helm chart could not be upgraded easily using helm (even though the API versions were corrected now) - it involved manually editing helm config map statuses to complete the upgrade.

Conclusion

This was one of the more complicated upgrades. Reading the changelog and provided upgrade instructions definitely helps.

More information here:

Labels:

Comments

Post a Comment

Popular posts from this blog

Manual Kubernetes TLS certificate renewal procedure

Intro Kubernetes utilizes TLS certificates to secure different levels of internal and external cluster communication.  This includes internal services like the apiserver, kubelet, scheduler and controller-manager etc. These TLS certificates are created during the initial cluster installation and are usually valid for 12 months. The cluster internal certificate authority (CA) certificate is valid for ten years. There are options available to automate certificate renewals, but they are not always utilised and these certs can become out of date. Updating certain certificates may require restarts of K8s components, which may not be fully automated either. If any of these certificates is outdated or expired, it will stop parts or all of your cluster from functioning correctly. Obviously this scenario should be avoided - especially in production environments. This blog entry focuses on manual renewals / re-creation of Kubernetes certificates. For example, the api-server certificate below...
Post a Comment
Read more

Analysing and replaying MySQL database queries using tcpdump

Why There are situations where you want to quickly enable query logging on a MySQL Database or trouble shoot queries hitting the Database server in real-time. Yes, you can enable the DB query log and there are other options available, however the script below has helped me in many cases as it is non intrusive and does not require changing the DB server, state or configuration in any way. Limitations The following only works if the DB traffic is not encrypted (no SSL/TLS transport enabled). Also this needs to be run directly on the DB server host (as root / admin). Please also be aware that this should be done on servers and data you own only. Script This script has been amended to suit my individual requirements. #!/bin/sh tcpdump -i any -s 0 -l -w - dst port 3306 | strings | perl -e ' while(<>) { chomp; next if /^[^ ]+[ ]*$/;   if(/^(ALTER|COMMIT|CREATE|DELETE|DROP|INSERT|SELECT|SET|UPDATE|ROLLBACK)/i) {     if (defined $q) { print "$q\n"; }     $q=$_; ...
Post a Comment
Read more

Deprecating Networking Ingress API version in Kubernetes 1.22

  Intro Kubernetes deprecates API versions over time. Usually this affects alpha and beta versions and only requires changing the apiVersion: line in your resource file to make it work. However with this Ingress object version change, additional changes are necessary. Basics For this post I am quickly creating a new cluster via Kind (Kubernetes in Docker) . Once done, we can see which API versions are supported by this cluster (version v1.21.1). $ kubectl api-versions | grep networking networking.k8s.io/v1 networking.k8s.io/v1beta1 Kubernetes automatically converts existing resources internally into different supported API versions. So if we create a new Ingress object with version v1beta1 on a recent cluster version, you will receive a deprecation warning - and the same Ingress object will exist both in version v1beta1 and v1. Create $ cat ingress_beta.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:   name: clusterpirate-ingress spec:   rules:  ...
Post a Comment
Read more