EKS Anywhere., decoding the architecture.
This article is part of the EKS Anywhere series EKS Anywhere, extending the Hybrid cloud momentum | by Ambar Hassani | Apr, 2022 | Medium
You really must be a passionate enthusiast to delve under the hood! Not for the faint-hearted…
In plain words., EKS Anywhere is a set of comprehensive, yet simple open-source based orchestrated tooling that is used to deploy EKS Distro to create and manage EKS Anywhere clusters.
Don’t get jittery about it! Remember, installation of EKS Anywhere with your first production grade cluster (if prerequisites are met) is a mere 45-to-60-minute task. The subsequent clusters are merely 8–10 minutes each (for a 2 control + 2 worker + 3 etcd node) configuration.
Let’s first understand EKS Anywhere from a solution standpoint
Question: What is EKS Anywhere., and I have also come across the term EKS Distro. Are these different., or the same…. I am confused!!!
Let’s attempt to answer this through the below visual, where you can see that EKS Distro is the actual Kubernetes distribution that runs in the clusters, whereas EKS Anywhere is an orchestrated tooling packaged as a bundled software package created by AWS that is used to create EKS Distro based Kubernetes clusters
For the technology scuba and someone who likes to uncover the actual implementations, the below visual is really oversimplified and places EKS Distro as a separate method., which in my view is misleading!!!
In my own perspective and to best of my knowledge, I would re-paint the visual like this, placing EKS-Distro right in the center, where it’s really agnostic of the methods.
Most importantly, what needs to be highlighted is that EKS Distro can be self-deployed using community tools (KOPS, Kubeadm, Kubeinit) on a variety of underlying infrastructure., but the key consideration in treading this path would parallel an enlightened Kubernetes soul !!!
OR
using EKS Anywhere, the preferred and purpose built tool-chain built by AWS for on-premises installations of EKS Distro on vSphere environments (bare-metal support coming soon)
The EKS Anywhere project wraps Cluster API, various CLI and plugins (eksctl cli, anywhere plugin, kubectl, aws-iam-authenticator) and other methods, routines of downloading required artefacts and release bundles in a single package to ease the administrative burden of deploying EKS Distro. In summary., a rich and evolving software package that is a preferred way of deploying EKS Distro based clusters on-premises.
Wow quite a few heavy terms used: EKS Anywhere, EKS Distro, EKS Anywhere control plane, Bootstrap cluster, CAPI/CAPV-Cluster API, Standalone clusters, Management cluster, Workload cluster.
DON’T WORRY., we will dig deeper into that later on.
For now, understand that AWS has open-sourced EKS Distro and EKS Anywhere projects and both of these are available via GitHub. The beauty of EKS Distro is that it is the same Kubernetes distribution that runs on AWS EKS cloud service. Being open sourced, one can deploy EKS Distro in a variety of ways for a consistent Kubernetes distribution experience.
Question: Ok, now I understand a bit about EKS Anywhere and EKS Distro Can you tell me a bit more of how it is deployed.
A high-level understanding of the two topologies/models is explained in this video
EKS Anywhere deployments can occur in two patterns and AWS has nicely documented the differences in this URL
EKS Anywhere deployments can occur in two patterns
First pattern: Standalone clusters. Since this is a standalone cluster, the EKS Anywhere control plane would be a combination of the components running on EKS Administrative machine plus the Cluster API components running on the standalone clusters.
Second pattern: Dedicated management cluster model. An interesting callout for this model would be that the term “management cluster” is more notional in nature. It is actually a standalone cluster referenced as management cluster that further acts to manage other workload clusters. The difference between the management (or standalone cluster) and the workload cluster is the presence of the cluster API components.
EKS Anywhere, Under the Hood!
The first and primary thought: Kubernetes clusters are a set of nodes or virtual machines stacked up with all the software required to run Kubernetes. So how are these clusters defined, configured, etc. in the context of EKS Anywhere. In that stride it’s quintessential to at-least understand the overview of Cluster API framework.
The video provides more than peep into the overall interactions of various build tooling in EKS Anywhere framework
Welcome Cluster API, a comprehensive method to building Kubernetes clusters using the declarative Kubernetes approach itself. In this framework, the cluster API software leverages a set of machine templates written in the same declarative style used by Kubernetes. And then., it communicates with the infrastructure provider APIs to create these machines intended to be the Kubernetes nodes based on these machine templates.
The cluster and machine templates themselves are a resource definition of sorts including compute, storage, network, operating system, etc. configurations that define the cluster nodes. In addition, the templates also consist of methods to talk to the infrastructure provider.
Note: Cluster API is a broader framework supporting major infrastructure providers, however in the context of EKS Anywhere this list may be limited to the specific integrations (e.g., vSphere, Apache Cloudstack) for virtualized IaaS
The below figure is straight out of Cluster API handbook and basically gives a high level overview of a parent task for creating clusters
Once the machine templates are validated and rendered by the underlying infrastructure provider, a series of ongoing reconciliations and health checks are performed. In addition to that, changes in the form of resource requests are monitored and deployed for existing clusters. These actions could be scale, upgrade or delete these machines, nodes via a change in the template. Now let’s do a quick peep into the Cluster API itself. Let’s reason this as a software package itself that constitutes of all the magic of doing the above.
Question: How and where do I run this Cluster API software? Simple answer is that Cluster API can run on any compliant Kubernetes cluster. However, to avoid a chicken and egg situation, the cluster API can be bootstrapped using a KinD (Kubernetes in Docker) cluster!
Most importantly you do not have to manually install Cluster API in EKS Anywhere, it’s embedded as a part of the build tooling.
If you haven’t read or are not familiar with Cluster API framework., then the below URL can be a good starting point. However, for now I would suggest that you skip this read and come back when you have some practical hands on with deploying EKS Anywhere., else the tendency to procrastinate will be higher.
Introduction — The Cluster API Book (k8s.io)
Now back to our topic and some more obvious questions!!!
When the command to deploy a cluster is executed for a standalone workload or a dedicated management cluster, a temporary KinD cluster known as a bootstrap cluster is automatically created on the administrative machine. This bootstrap KinD cluster then uses the inbuilt Cluster API routines that instructs the vSphere to create the cluster nodes.
You can see from the process of cluster creation for Standalone workload or a management cluster and observe what’s going on
Performing setup and validations
✅ Connected to server
✅ Authenticated to vSphere
✅ Datacenter validated
✅ Network validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Control plane and Workload templates validated
Provided VSphereMachineConfig sshAuthorizedKey is not set or is empty, auto-generating new key pair...
VSphereDatacenterConfig private key saved to w01temp/eks-a-id_rsa. Use 'ssh -i w01temp/eks-a-id_rsa capv@<VM-IP-Address>' to login to your cluster VM
✅ Vsphere Provider setup is valid
✅ Create preflight validations pass
Creating new bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific setup
Creating new workload cluster
Installing networking on workload cluster
Installing storage class on workload cluster
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing AddonManager and GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!The below video also describes the above process in a deeper narrative via the verbose level 9 execution logs for cluster creation.
If you are a serious deep-diver and want to understand more
- EKS Anywhere release manifest: This release bundle gets downloaded everytime one creates a new standalone workload cluster or management cluster. The manifest is the entry point to everything else that happens in creating the EKS Anywhere cluster/s
- Specific artifacts used by EKS Anywhere: Visit the page eks-anywhere/artifacts.go at main · aws/eks-anywhere (github.com) and one can view the v1alpha1 version of artifacts bundled within EKS Anywhere
- EKS Anywhere Build tooling: go to EKS-Anywhere-build-tooling. The URL should lead to the README section of Base image track, and you can select “Click to view/hide table”. It should give you an indicative view of what tools are used to create the artefacts for EKS Anywhere.
- EKS-Distro release bundles: If one is interested in gaining further knowledge, you can check here EKS Distro (amazonaws.com). Click on the individual releases to see the release manifest and the components included.
OK., now on to the cluster constructs, what runs inside our EKS Anywhere clusters
Depending on the role of the node (control/worker/etcd) and the type of cluster (standalone/management/managed workload), the functional components will be rendered as PODS in different namespaces.
Please increase the browser zoom to see which PODS are running in the respective nodes.
Let’s unravel some additional points
Operating system used by EKS Anywhere cluster nodes: Bottle-rocket (AWS developed OS) or a standard Ubuntu 20.04 OS image or a custom Ubuntu 20.04 OVA image. There are good number of reasons why one would prefer one versus other. However, the primary reason of using ubuntu would be to customize the OS via additional software packages or bring in self-signed certificates since bottle-rocket does not allow for any of those.
Bottle-rocket is automatically downloaded by EKS Anywhere from the AWS repositories while creating the first cluster itself. The ubuntu OS template needs to be built with a specific procedure and if required one can perform customizations.
Infrastructure provider: EKS Anywhere can run on vSphere 7+ and vSphere 6.7 u3 and above or even bare metal. There are ongoing developments and will continue such as EKS Anywhere on Apache cloudstack and more. Although my PoC environment works perfectly fine on vSphere 6.7 u3+, it is not recommended as AWS will not support it contractually.
Other dimensions: Lot of information available on EKS anywhere dedicated website Amazon EKS Anywhere | EKS Anywhere (amazonaws.com) in terms of preferred CNI, CSI, load balancers, ingress, Authentication mechanisms, etc. But the key point to note is that while the website states some specific tools, the EKS Anywhere partner ecosystem and the community driven use-cases are on the rise.
As an Example, the default CNI used by EKS Anywhere is Cilum., well does it mean I cannot use Calico. Of course, you can use Calico., but it could mean a complex arrangement of sorts from a support standpoint… see the blog from Calico team. Calico on EKS Anywhere (tigera.io)
Likewise, the preferred Ingress as listed on the EKS Anywhere website is Gloo Edge or Emissary-ingress., but that does not mean one cannot deploy NGINX ingress on EKS Anywhere clusters.
Deployment & Support model: Well, the only reference I could see is that AWS supports EKS Anywhere with a pre-requisite that customers should have AWS Enterprise support subscription. I am quite unsure of the partner plays here as the entire thing is open-sourced., so I guess this is an interesting playground for many services partners as the deployments increase. An important point to note is that when subscribed to AWS support, one can avail of add-ons in the form of curated packages for certain open-source
This brings us to a logical conclusion to the architectural decoding of EKS Anywhere. I know there is a lot to digest., but if you fancy the inner workings then hopefully this article along with the links included gives you a good headstart!
To be continued., if I find more useful information that can be added to this article
cheers,
Ambar@thecloudgarage
#iwork4dell
