Skip to main content

Building EKS (Amazon hosted Kubernetes) clusters using eksctl

 


Disclaimer

I have published this post on my work blog https://reece.tech previously.

Overview

Eksctl acts as a wrapper around CloudFormation templates. Creating a cluster will add one stack for the control plane (EKS master servers) and one stack for each node group configured (a node group is a group of workers using the same networking and sizing as well as IAM permissions).

However, certain actions such as upgrading the Kubernetes master or worker version or scaling out the number of workers in a node group does not always update the CF stacks associated with it.

Preparation

Download and install the latest version of eksctl.

Follow the Weaveworks installation guide: https://eksctl.io/introduction/installation/.

Download eksctl (Linux)

curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

Install eksctl (Linux)

sudo mv /tmp/eksctl /usr/local/bin

Provide AWS credentials

Ensure the AWS credentials are set for your current session. The easiest way is to run (aws configure) with the required region and access keys - however as things get more complicated (multiple accounts, assume role and mfa), additional scripts or applications may be required.

Planning

Our first EKS test cluster should be simple. Just running eksctl create cluster will create a new VPC, Internet Gateway, subnets and all other resources required for initial testing.

However, this cluster will be available externally (with EC2 instances / workers and the API server endpoint exposed to the internet). Ideally we would like to utilise existing networks and VPCs as well as adhering to existing security policies and regulations.

Assuming we decided on a setup with three internal subnets to be used for workers and (initially) an external EKS control plane / API server endpoint. We create a resource definition to be used with eksctl:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: myeks
  version: '1.13'
  region: ap-southeast-2
 
# Role used by the EKS control plane itself when managing resources
iam:
  serviceRoleARN: "arn:aws:iam::123412341234:role/basic/eks-cluster-service-role"
 
# Where to deploy the control plane endpoints, and the worker nodes
vpc:
  id: vpc-12341234123412345
  subnets:
    private:
      ap-southeast-2a: { id: subnet-12341234123412345 }
      ap-southeast-2b: { id: subnet-12341234123412346 }
      ap-southeast-2c: { id: subnet-12341234123412347 }

nodeGroups:
  - name: mynodes
    instanceType: r4.large
    desiredCapacity: 3
    privateNetworking: true
    securityGroups:
      withShared: true
      withLocal: true
    iam:
      instanceProfileARN: "arn:aws:iam::123412341234:instance-profile/basic/eks-cluster-iam-NodeInstanceProfile-1PSA1WKT5RP16"
      instanceRoleARN: "arn:aws:iam::123412341234:role/eks-cluster-node-instance-role"
    ssh:
      allow: true
      publicKeyPath: test

Creation

eksctl create cluster --config-file config.yaml --kubeconfig $HOME/.kube/config.eks

This takes around 12 minutes after which the worker nodes are running and showing as Ready in K8s:

$ kubectl get nodes -o wide
NAME                                               STATUS   ROLES    AGE   VERSION              INTERNAL-IP     EXTERNAL-IP     OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-192-168-12-18.ap-southeast-2.compute.internal   Ready    <none>   4m    v1.13.7-eks-c57ff8   192.168.12.18   52.64.78.71     Amazon Linux 2   4.14.123-111.109.amzn2.x86_64   docker://18.6.1
ip-192-168-40-73.ap-southeast-2.compute.internal   Ready    <none>   4m    v1.13.7-eks-c57ff8   192.168.40.73   13.211.179.67   Amazon Linux 2   4.14.123-111.109.amzn2.x86_64   docker://18.6.1

Scaling

We can easily scale the cluster out and back in again.

eksctl scale nodegroup --cluster=myeks --name=mynodes --nodes=6
eksctl scale nodegroup --cluster=myeks --name=mynodes --nodes=2

Upgrades

Upgrading only upgrades to the next available higher version, so no version info is necessary.

eksctl update cluster --name=myeks --approve

To upgrade the workers / node pools, simply create a new pool and remove the existing one after.

eksctl create nodegroup --config-file config.yaml # create new node group
eksctl delete nodegroup --config-file config.yaml --only-missing # remove old node group

Deletion

eksctl delete cluster --name=myeks
Labels:

Comments

Post a Comment

Popular posts from this blog

Enhancing Kubernetes Security: Enabling API Server Audit Logging

      In the ever-evolving landscape of container orchestration and management, Kubernetes remains at the forefront, empowering organizations to deploy, manage, and scale their containerized applications efficiently. However, as Kubernetes environments grow in complexity, ensuring robust security measures becomes paramount. One crucial aspect of Kubernetes security is audit logging, particularly concerning the Kubernetes API server, which serves as the primary point of interaction with the cluster. The Importance of Audit Logging Audit logging provides visibility into the activities and interactions within a Kubernetes cluster. By recording requests and responses made to the API server, audit logs offer valuable insights into user actions, system modifications, and potential security breaches. This transparency is essential for compliance with regulatory requirements such as HIPAA, GDPR, and PCI DSS, as well as for detecting and investigating security incidents. Understan...
Post a Comment
Read more

Manual Kubernetes TLS certificate renewal procedure

Intro Kubernetes utilizes TLS certificates to secure different levels of internal and external cluster communication.  This includes internal services like the apiserver, kubelet, scheduler and controller-manager etc. These TLS certificates are created during the initial cluster installation and are usually valid for 12 months. The cluster internal certificate authority (CA) certificate is valid for ten years. There are options available to automate certificate renewals, but they are not always utilised and these certs can become out of date. Updating certain certificates may require restarts of K8s components, which may not be fully automated either. If any of these certificates is outdated or expired, it will stop parts or all of your cluster from functioning correctly. Obviously this scenario should be avoided - especially in production environments. This blog entry focuses on manual renewals / re-creation of Kubernetes certificates. For example, the api-server certificate below...
Post a Comment
Read more

Choosing the Right Operating System for Kubernetes

  In the realm of container orchestration, Kubernetes reigns supreme as the de facto standard. It's the go-to platform for managing containerized applications at scale. However, as you delve into the world of Kubernetes deployment, one crucial decision often overlooked is the choice of operating system (OS). While Kubernetes itself is platform-agnostic, the underlying OS can significantly impact performance, security, and overall manageability. In this guide, we'll explore various operating systems suitable for Kubernetes deployment and help you make an informed decision. 1. Linux Distributions: a. Ubuntu: Pros: Ubuntu is one of the most popular choices for Kubernetes deployment due to its widespread adoption and strong community support. It offers a balance between stability and cutting-edge features. Cons: Some argue that Ubuntu's release cycle may introduce instability, especially for production environments requiring long-term support (LTS) versions. b. CentOS/RHEL: P...
Post a Comment
Read more