EKS Anywhere., AWS console integration
This article is a part of the multi-part story of EKS Anywhere. Access it here EKS Anywhere, extending the Hybrid cloud momentum | by Ambar Hassani | Apr, 2022 | Medium
Motivation for this article: EKS Anywhere is a relatively new offering and as a consequence, the official documentation and the internet blog posts are still evolving.
One of the key pursuit for anyone deploying EKS Anywhere clusters could be a possible integration with EKS console.
In that context, there are a number of steps which need to be performed and the official documentation personally for me little-too generic and obscures the specificity of security and IAM.
At the same time the documentation is spread too thin between bread-crumbs of console-based, aws-cli based and cloud-formation methods rather than an end-to-end integration via the single eksctl cli that is generally used for all things EKS (including EKS Anywhere).
I have come across blog posts where the authors have also noted they had tripped over this aspect many-a-times before having a successful go at something that is so easy and important in the overall end game of EKS Anywhere.
What can I expect at the end of this article. The main goal of this article is to demonstrate how you can easily observe the EKS Anywhere clusters using the EKS console within AWS with the help of EKS connector.
Please note that as of this writing EKS console provides a read-only but very rich and comprehensive view into your EKS Anywhere clusters.
It does not allow any write-operations, like scaling clusters, applying policies or deploying workloads, etc.
Also note that EKS console is not the only way to observe and manage your clusters, TANZU Mission Control, Azure ARC, Nirmata, etc. are all viable and possible methods.
Then the obvious question would be WHY SHOULD I INVEST MY TIME AND EFFORT to make this come true., when it’s just a read-only console.
#1 It’s practically free (except maybe very abysmal costs of ssm) giving you a comprehensive drill-down into your clusters, nodes, workloads, etc.
#2 If your EKS ecosystem is hybrid, i.e. you have clusters on Managed EKS, it gives you a single-window in to all your cloud and on-premises/co-located clusters
# Knowing AWS, it’s could be a start into many more richer capabilities into the future roadmap of EKS hybridity., but no comments for now!
An exemplified view looks like the visuals below.
Each of the sub-items, whether the nodes or workloads are clickable to show further drill-down information allowing a good range of manageability and observations.
A visual interpretation of EKS Anywhere Connector, which has been open-sourced by AWS. We will observe greater detail during the course of the article.
Pre-requisites:
- An IAM user with console and programmatic access privileges along with the permissions documented via this JSON policy.
- An operational EKS Anywhere cluster that needs to be viewed in the EKS console.
This might seem a little overwhelming to begin, however that is not the case. So stay calm! and follow the steps.
Step-1 Setting up the IAM user with appropriate permissions
Assume that you are an EKS-Anywhere administrator and are tasked with the responsibility of deploying the on-premises clusters. At the same time, you have also been asked to integrate and manage these clusters via the EKS console.
As a result, it’s obvious that you will need an IAM user for yourself in your company’s AWS account. And since your scope is only limited to EKS Anywhere administration., as per principle of least privilege your account will need to ring-fence the permissions model.
So the first question., what is going to the permissions policy for your IAM user
You will need to ask your AWS admin who is in-charge of creating user accounts to create your IAM user identity and enable it with the permission policy located here https://raw.githubusercontent.com/thecloudgarage/eks-anywhere/main/eks-connector/eksa-admin-user-IAM-policy.json
Note: The above policy is an aggregation of various scattered methods documented in AWS official sites. Although it has been trimmed to scope in the relevant permissions for EKS-Anywhere integrations within EKS console, there could be further fine-graining that can be performed.
Step-2 Register and Deregistering your clusters along with viewing them in the EKS console
Once done., and you have received your IAM user id, password and programmatic access keys., you will need to execute a single script to integrate the EKS Anywhere cluster with EKS console.
I have created this script to ease the administrative burden of running multiple commands. The script is already made available in the EKS Anywhere Administrative machine when it was installed as per this article
To start, SSH into the EKS Anywhere administrative machine as user ubuntu
Keep your AWS IAM credentials (created above) handy before executing this script
cd $HOME
source $HOME/eks-anywhere/cluster-ops/register-cluster.shYou should see a similar output as shown below
Login to the AWS console with IAM user that was created for you and navigate to the EKS service > clusters in the region specified.
You should be able to view the registered cluster and its objects.
And now if you want to “deregister your cluster”., then simply download the deregister script and execute it. It will de-register the cluster, remove all the IAM roles and Kubernetes RBAC deployed on your cluster.
cd $HOME
source $HOME/eks-anywhere/cluster-ops/deregister-cluster.shThat’s it… a short walk., we are done. Hope you enjoyed reading it!
cheers
Ambar@thecloudgarage
The below reading is OPTIONAL only if you want to break your head as to what is happening under-the-hood or behind the scenes
To start understanding the architectural nuances and various functional components, let’s visualize the entire affair as a set of workflows along with their implementation details.
Image credits: Connect any Kubernetes cluster to Amazon EKS | Containers
First workflow
Second workflow
Third workflow
As one can observe from the three workflows, it’s pretty self-explanatory that the interactions between AWS and EKS Anywhere clusters require multiple levels of permissions. These permissions are required at both AWS end in the form of IAM roles and policies and within the EKS Anywhere cluster in the form of RBAC roles and bindings.
There are 4 noteworthy permission sets that are “AUTOMATICALLY CREATED AND DEPLOYED” as a part of EKS connector setup.
Let’s name them in the context of the functions they perform.
- AmazonServiceRoleForAmazonEKSConnector. This is an AWS IAM service-linked role and is automatically created by the eksctl command during the script run. Note, that this role creation is done only once during the registration of any EKS Anywhere cluster.
- Amazon EKS Connector Agent role: This is an AWS IAM service role, which is automatically created & applied during script execution. This role is created during the cluster registration process and automatically torn down during the cluster deregistration process. You can observe these roles in your IAM console as eksctl-xxxxxxxxxxxxxx meaning that these were created by eksctl cli as a part of cluster registration process.
- eks-connector-console-dashboard-full-access-group. This is a k8s cluster role and bindings RBAC which gets automatically created and applied on EKS Anywhere cluster during script execution. This is a YAML consisting of cluster roles and bindings for the cluster to be registered.
- eks-connector-clusterrole This is a k8s cluster role and bindings RBAC which gets automatically created and applied on EKS Anywhere cluster during script execution. This is a YAML consisting of cluster roles and bindings for the cluster to be registered.
Let’s start detailing each of these…
AmazonServiceRoleForAmazonEKSConnector (AWS IAM service-linked role): Most notably, one needs to understand that the EKS service is coordinating various interactions for successful registration of the EKS Anywhere cluster.
These include EventBridge and Systems Manager services. Amazon Elastic Kubernetes Service uses the AWS Identity and Access Management (IAM) service-linked roles for this purpose.
A service-linked role is a unique type of IAM role that is linked directly to Amazon EKS. Service-linked roles are predefined by Amazon EKS and include all the permissions that the service requires to call other AWS services on your behalf
In this case, it includes managing activation in Systems Manager for the EKS Connector agent and for receiving events from EventBridge whenever the EKS Connector agent is registered and deregistered with the Systems Manager service. Hence, it’s obvious that the EKS service itself will need a Service-linked role to perform these interactions.
EKS Connector agent role (AWS IAM service role): This is an IAM service role (not a service-linked role) and in this case is automatically created when eksctl register command is run inside the register-cluster.sh script.
The Amazon EKS Connector is based on the System Manager (SSM) Agent, which is also open-sourced like the EKS connector. This IAM role allows the EKS connector agent to interact with SSM service
To understand the significance of this IAM service role, it is important to understand how the EKS connector functions along with AWS services., most importantly the SSM service
The EKS Connector runs as a StatefulSet on your Kubernetes cluster. It establishes a connection and proxies the communication between the API server of your EKS Anywhere cluster and Amazon Web Services. It does this to display cluster data in the Amazon EKS console until you disconnect the cluster from AWS.
A manifest file which we will talk later, deploys all the components to setup the EKS connector in your EKS Anywhere cluster. It mainly consists of three functional components
InitContainer: This container registers the EKS Connector agent with the Systems Manager control plane service and persists the registration information in the Kubernetes backend data store. InitContainer mounts this data to the EKS Connector agent’s volume when it is recycled. This eliminates the need of registration whenever a pod is recycled.
EKS Connector agent: This is an agent based on the SSM agent, running in container mode. This agent creates an outbound connection from the Kubernetes cluster to the AWS network. All subsequent requests from AWS are performed using the connection channels established by the EKS Connector agent.
Connector proxy: This agent acts as a proxy between the EKS Connector agent and Kubernetes API Server. This proxy agent uses the Kubernetes service account to impersonate the IAM user that accesses the console and fetches information from the Kubernetes API Server.
As one can see that EKS connector agent (based on SSM agent) connector liaisons with the SSM service, which in turn interacts with EKS service via EventBridge. To facilitate these interaction, the EKS connector agent role is required with appropriate permissions to create, open and control the SSM channels. In absence, of this important IAM role at AWS end, the creation and control of Systems Manager channels would not be possible eventually leading to an unsuccessful registration
Upon successful registration, one can notice the changes in the AWS EventBridge services. A new event rule with the pattern of registration and deregistration is create under the “default” event bus.
Eks-connector-console-dashboard-full-access-group (k8s cluster role and bindings RBAC on EKS Anywhere cluster) This RBAC set is referenced as eks-connector-console-dashboard-full-access-group.yaml in our implementation steps.
In our case, this RBAC set is automatically generated via the eksctl register command running inside the register-cluster.sh script. Later, the YAML is automatically applied on the EKS Anywhere cluster as a part of script execution
The significance of this RBAC on EKS Anywhere cluster is to facilitate AWS IAM users with different levels of authorization to access the Kubernetes cluster.
This is done through ClusterRole and ClusterRoleBinding deployed in the EKS Anywhere cluster. One can observe the pattern in the below YAML, wherein the “get” and “list” verbs (actions) are permitted for various resources. This YAML creates the necessary cluster roles and bindings within the EKS Anywhere cluster.
In the below template, the variable “%IAM_ARN%” refers to the ARN of the actual AWS IAM users trying to access the EKS Anywhere cluster information (nodes, workloads, etc.) from the EKS console
Eks-connector-clusterrole (k8s cluster role and bindings RBAC on EKS Anywhere cluster): This RBAC set is referenced as eks-connector-clusterrole.yaml.
EKS Connector uses user impersonation mechanism to authorize against Kubernetes API server. This set of RBAC on EKS Anywhere cluster allows the ServiceAccount used by the proxy agent to impersonate a particular AWS IAM user through it’s permitted ARN.
When an IAM user performs an action on the EKS Console to view Kubernetes objects, their IAM ARN along with the Kubernetes requests are sent to the EKS Connector agent through the Systems Manager service. Once the EKS Connector agent receives the request, it connects to the local Unix socket to pass the Kubernetes requests and IAM ARN to the proxy agent.
The proxy agent uses the mounted ServiceAccount token to authenticate and it uses the IAM user ARN for authorization to fetch the information from Kubernetes. This ensures the IAM users can only access the objects that are allowed by Kubernetes RBAC.
The narrative can be observed in the below YAML template, which sets up the cluster roles and bindings for the EKS connector. The verb “impersonate” along with resources “users” and resourceNames “%IAM_ARN%” combine to create the necessary permission set.
In our case, this RBAC set is automatically generated via the eksctl register command running inside the register-cluster.sh script. Later, the YAML is automatically applied on the EKS Anywhere cluster as a part of script execution
Now for some quick observations! Once the registration is fully completed and you are able to view the cluster, nodes, workloads, head over to the EKS administrative machine and issue the below command to get detailed information depending on the node configurations in your cluster
Hopefully by now, you would have got a good comprehension of the Why, What and How of these interactions along with the permissions required to securely setup the EKS console access for the EKS Anywhere clusters.
cheers
Ambar@thecloudgarage
#iwork4dell
