AWS EKS: Auth optimization for K8s
Amazon Elastic Kubernetes Service (EKS) makes it easy to deploy, manage, & scale containerized applications using Kubernetes on AWS.
EKS uses Identity & access management (IAM) to provide authentication to your Kubernetes (K8s) cluster through the AWS IAM Authenticator for Kubernetes, but it still relies on native K8s Role-Based Access Control (RBAC) for authorization.
This means that IAM is only used for authentication of valid IAM entities. All permissions for interacting with your EKS cluster’s K8s APIs are managed through the native K8s RBAC system.
You can configure the stock kubectl client to work with EKS by installing the AWS IAM Authenticator for K8s & modifying your kube/config file to use it for authentication.
Here’s the AWS official documentation on how to onboard users to EKS. This approach is not scalable if we have a large set of users who require a similar set of permissions. This will generate a long list of users to maintain in the EKS auth ConfigMap & makes EKS operations cumbersome.
We can solve this problem by utilizing AWS IAM groups, roles & EKS auth ConfigMap. This will help you efficiently manage EKS authentication from a single pane.
Before we begin, let’s discuss the problem we are trying to solve: Let’s assume we are an organization having about TEN EKS administrators who manage our geographically distributed cluster.
Now using the AWS prescribed method, which is the IAM user to K8s group mapping, the auth ConfigMap(aws-auth.yaml) will look something like this:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapUsers: |
- userarn: arn:aws:iam::555555555555:user/admin1
username: admin1
groups:
- system:masters
- userarn: arn:aws:iam::555555555555:user/admin2
username: admin2
groups:
- system:masters
- userarn: arn:aws:iam::555555555555:user/admin3
username: admin3
groups:
- system:masters
...
...
... - userarn: arn:aws:iam::555555555555:user/admin10
username: admin10
groups:
- system:masters
As we can see, the file can grow very quickly as more users are on-boarded. Moreover keeping a 1:1 map of IAM users (of a similar type) in a ConfigMap is not the most efficient.
Disadvantages with this approach:
- Difficult to add/remove users as editing a ConfigMap containing 1000 lines is not intuitive
- Easy to commit mistakes while searching/editing entries in ConfigMap
- Two different places to make modifications (AWS IAM, K8s auth ConfigMap) each time a user has to be on-boarded/removed
- There is a limit on how big a ConfigMap can grow (currently it is 1 MB, the max object size etcd can store). This approach will not work if you have a large set of users in your organization
Hence, it’s more efficient to provide two more layers of abstraction (IAM groups & roles) to K8s through the use of IAM user to IAM group to IAM role to K8s group mapping in AWS
Now, let’s see how this new approach helps us & below are the steps to implement this auth abstraction:
You will need to set up IAM roles, policies, groups & users either from the AWS console, CLI, or through an orchestration tool like Cloudformation, Terraform, etc.
- The first step is to set up an IAM role that K8s can refer
- Navigate to AWS Console > IAM > Roles > Create role
- Create a role, we can call it “eks-admin-role”. This role need not have any policies attached to it but will need a trust relationship
- Under the Trust Relationship tab, click on Edit trust relationship & fill in the below policy statement. This policy will allow any user in the account to assume this role, but we will restrict it to a specific group by attaching the assume role policy to the admin group
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AWS-ACCOUNT-ID>:root"
},
"Action": "sts:AssumeRole"
}
]
}2. In this step, we will create the necessary IAM policies that an EKS admin would require
- Navigate to AWS Console > IAM > Policies > Create policy
- Create a policy called “eks-admin-access”, which will grant CRUD (Create, read, update, delete) access to EKS
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "eks:*",
"Resource": "*"
}
]
}- In addition to the above policy, create one more policy called “eks-admin-assume” that will permit an admin group, the permission to assume the role (“eks-admin-role”) that was created in Step1
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::<AWS-ACCOUNT-ID>:role/eks-admin-role"
]
}
]
}3. Here, let’s create an IAM admin group called “EKS-admins” & attach the previously created policies (“eks-admin-access” & “eks-admin-assume”). This will allow users who are a part of the admin group to have CRUD permissions to EKS resources & can also assume the admin role to perform superuser operations in K8s
- Navigate to AWS Console > IAM > Groups > Create New Group
4. Next, we will create an IAM admin user called “admin1” & make it a part of the admin group created in the previous step
- Navigate to AWS Console > IAM > Users > Add User
5. Install kubectl, awscli & aws-iam-authenticator on the machine from which you will interact with K8s APIs
- The IAM user who created the EKS cluster will have access to edit
aws-auth.yamlConfigMap. Open the ConfigMap & add the lines in the below section which are in bold. This will add the IAM role (“eks-admin-role”) to K8s cluster-admin group, ie. system:masters
---
apiVersion: v1
data:
mapRoles: |
- rolearn: arn:aws:iam::<AWS-ACCOUNT-ID>:role/worker-nodes
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::<AWS-ACCOUNT-ID>:role/eks-admin-role
username: eks-admin
groups:
- system:masters- Once done, create AWS Access Keys (Access Key ID & Secret Access Key) for the IAM admin user “admin1”. Configure AWS CLI with the new credentials
$ aws configureAWS Access Key ID [****************TUBR]:
AWS Secret Access Key [****************as3w]:
Default region name [us-west-2]:
Default output format [json]:
- Verify that the new admin user “admin1” can assume the “eks-admin-role” role by entering the below commands
$ aws sts get-caller-identity{
"UserId": "****************TUBR",
"Account": "<AWS-ACCOUNT-ID>",
"Arn": "arn:aws:iam::<AWS-ACCOUNT-ID>:user/admin1"
}$ aws sts assume-role \
--role-arn arn:aws:iam::<AWS-ACCOUNT-ID>:role/eks-admin-role \
--role-session-name test-session{
"Credentials": {
"AccessKeyId": "****************TUBR",
"SecretAccessKey": "****************as3w",
"SessionToken": "<TOKEN>",
"Expiration": "2019-09-05T21:36:58Z"
},
"AssumedRoleUser": {
"AssumedRoleId": "<ASSUME-ROLE-ID>:test-session",
"Arn": "arn:aws:sts::<AWS-ACCOUNT-ID>:assumed-role/eks-admin-role/test-session"
}
}
6. Finally, open the kube/config file (located at ~/.kube/config) & fill in the values which are in bold. The RoleARN is the ARN of the IAM admin role (“eks-admin-role”)
$ vi ~/.kube/config---
apiVersion: v1
clusters:
- cluster:
server: <EKS-ENDPOINT-URL>
certificate-authority-data: <BASE64-ENCODED-CA-CERT>
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: aws
name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "<EKS-CLUSTER-NAME>"
- "-r"
- "arn:aws:iam::<AWS-ACCOUNT-ID>:role/eks-admin-role"
- Validate that you have admin permissions in the K8s cluster
$ kubectl auth can-i delete configmaps -n kube-system
yesTo make another user an EKS admin, simply add the IAM user to the IAM admin group (“EKS-admins”) & the user will automatically be added to “system:masters” cluster-admin group in K8s without having to really make any changes in K8s
Similarly, it’s possible to create multiple IAM groups based on the permissions they would require (viewer, auditor, engineer, etc) & add users to these groups accordingly.
Hope this article helps to manage your infrastructure more efficiently!!!
REFERENCES
- https://docs.aws.amazon.com/eks/latest/userguide/managing-auth.html
- https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
- https://kubernetes.io/docs/reference/access-authn-authz/rbac
Join FAUN!
Subscribe to FAUN topics and get your weekly dose of the must-read tech stories, news, and tutorials 🗞️
Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬
