K8s Authentication and Authorization

Introduction

Security becomes the most important part when you are working with server clusters. Using encrypted communication, limiting access to non-admin users these thing needs to implemented without any compromise. Like any other security setup K8s also uses Authentication and after that Authorization to implement that in cluster. Before discussing that some basic SSL/TLS concept is needed to be understood. How the client and server interacts with each other, how they verifies themselvs and then their communication starts.

Basic of SSL/TLS

To complete the SSL/TLS some key exchange is needed. This is the basic setup,

  1. Client will have a symetric key/session key (Private only to client initially), and a certificate that validates the client
  2. Server have one Private key, one Public key(called Asyemetric key), and a certificate that validates the server.
  3. A certificate authority (CA) to sign the certificates to them.

Here discussing two key exchange techniques,

  1. Classic Key exchange Technique, where actutal key exchange happens.
  2. Mordern Key exchange technique, where they securely derive a shared symmetric session key.

Classic Key exchange Technique

When a client connects to a server, the server first returns its asymetric public key and certificate. Then the client verifies the certificate from the certificate authority and sends a randomly generated symmetric session key to the server after encrypting with the servers public key. Then the server decrypts clients symmetric session key with its private asymetric key. Now both server and client have a common private key to make further communication.

Mordern Key exchange technique

When a client connects to a server, the server sends its public key and certificate. The client verifies the certificate. The client and server then perform a key exchange (such as Diffie-Hellman) to securely derive a shared symmetric session key without sending it directly. This symmetric key is then used to encrypt further communication.

Authentication and Authorization process

That was how generally key exchange SSL/TLS verification happens. In kubernetes whenever this key or certificate things comes the does the same thing as inside every system resource like kubelet, kube-apiserver, kube-proxy, etcd communicates with each other. Authentication and authorization is needed for K8s users as not every users should be given Admin previlages. First new user will be authenticated by the kube-apiserver then admin gives authorization to the user as required for that user.

This user can be a human user or a non-human/bot user. This non-human/bot user is called Service Account. When a pod wants to comunicate to the api-server for tasks like reading other pod's status, using dynamic config-map files in this cases that pod must be tied to service account, so that api-server can allow it do this kind of tasks.

Authentication

Now let's discuss about human users like someone named Alice. Alice is a new user to whom we have to give some limited access of the cluster. This process starts from creating a private by Alice. After that with this private key Alice will generate certificate signing request(csr). This csr will be appproved by the admin. After approving the csr you will get a certificate. Now this certificate is Alice's user certificate. With this user certificate admin will modify the kube-config file with Alice's username, certificate and context(combination of username and cluster-name for limiting access). This kube-config is what we always pass by default with kubectl command. Modifying the the kube-config file will let the kube-apiserver authenticate the new user. Now admin will create a new context and Alice will switch to that. This completes the authentication part.

Authorization

Now how much access should we give to Alice is defined via authorization configuration. This is done with creating two files, Role and Role-binding file for namespaced resources and Cluster-role and Cluster-role-binding file for cluster wide resources. This role file will define what kind or how much access we will give to Alice and role-binding file will bind that role to the user. In this way Authorization is done for the user.

Mostly used types of Authentication,

  1. certificates,
  2. bearer tokens,
  3. service account tokens,
  4. OIDC (OpenID Connect),
  5. webhooks.

Mostly used types of Authorization,

  1. RBAC (most common) Role-based access control
  2. ABAC Attribute-based (legacy)
  3. Node special permissions for kubelets
  4. Webhook external authorization system
  5. AlwaysAllow / AlwaysDeny

Here while creating Alice user we have used authentication type as certificates and authorization type as RBAC. You can read about ABAC and its deference with RBAC here: https://rakeshbajpayee.in/posts/K8s_ABAC_and_RBAC.

For this authorization and authentication, every K8s manifest file and certificate lives in the node's control plane. In this directory /etc/kubernetes/manifests you will kube-apiserver.yaml file, and inside of that you will find clusters, users, and contexts, and you can also define the certificates related to each user. And in the container spec, you will find kube-apiserver will pass a few commands inside the container about authorization modes also. And all the certificates will also live inside /etc/kubernetes/pki the directory.

Now when we are passing the kubectl command, by default one kubeconfig file is also passed. On that file we are also specifying those clusters, users, contexts, and the certificates for the users. Now this file is verified by the kube-api server and lets the user or admin user use the cluster. And in case of multiple clusters, we can also pass multiple kubeconfig files via these commands.

While authenticating we are working with the kube-config file and while authorizing working with the kube-apiserver manifest file.

Example Kubeconfig File

apiVersion: v1
kind: Config

clusters:
- name: my-cluster
  cluster:
    server: https://192.168.1.100:6443
    certificate-authority: /etc/kubernetes/pki/ca.crt

users:
- name: admin-user
  user:
    client-certificate: /etc/kubernetes/pki/admin.crt
    client-key: /etc/kubernetes/pki/admin.key

contexts:
- name: admin-context
  context:
    cluster: my-cluster
    user: admin-user

current-context: admin-context

Example kube-apiserver manifest file

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - name: kube-apiserver
    image: k8s.gcr.io/kube-apiserver:v1.29.0
    command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-account-issuer=https://kubernetes.default.svc

Using single kubeconfig

export KUBECONFIG=$HOME/.kube/config

Using multiple kubeconfig files

You can chain multiple kubeconfig files using a colon.

export KUBECONFIG=$HOME/.kube/config:$HOME/.kube/dev.yaml:$HOME/.kube/prod.yaml

Here the order matters, as later files can override earlier ones. After this, kubectl treats them as one merged config.

Authentication steps

New user will create a private key, and based on that private key user will create a csr(certificate signing request).

command to create private key

openssl genrsa -out KEY-NAME.key 2048

command to create csr

openssl req -new -key KEY-NAME.key -out KEY-NAME.csr -subj "/CN=USER-NAME"

Now admin will take the CSR from the user, create a csr.yaml file and inside that csr.yaml file spec.request: section the csr will be given. One thing before putting that csr value on that, which is it needs to be done after converting to base64 format. command to do that

cat CSR-NAME.csr | base64 | tr -d "\n"

Output of this have to pass as value to that spec.

After this csr object needs to be created with

kubectl apply -f csr.yaml
kubectl get csr (to view the csr)

Now this csr needs to be approved by autorized CA, but admin can also approve that using,

kubectl certificate approve CSR-NAME

By approving this CSR it will now generate a certificate to that user and this certificate can be used in kubeconfig file to assign certian roles to that user. Again you need to decode this to base64 before using in kubeconfig.

command to check validity of certificate

openssl x509 -noout -dates -in CERTIFICATE-NAME.crt
kubectl get csr CSR-NAME -o yaml > NEW-YAML.yaml

here you will again get the certificate in spec.request field.

decoding this to base64, bash SPEC.REQUEST-VALUE | base64 -d

With this now the user have issued a certificate approved by the admin and ready to be used in kubeconfig file to give access to the user.

Authorization steps

kubectl auth can-i get pod

kubectl auth whoami

kubectl auth can-i get pod  --as USER-NAME

kubectl get roles -A --no-headers | wc -l           #(to count total number of roles)

Now to login as new user need to execute the following commands to make the entry in kubeconfig

kubectl config set-credentials USER-NAME --client-key=USER-PRIVATE-KEY.key --client-certificate=CERTIFICATE-NAME.crt

kubectl config view            #(To view the current kubeconfig file)

Now need to setup the context

kubectl config set-context CONTEXT-NAME --cluster=CLUSTER-NAME --user=USER-NAME   # (To create a new context)

kubectl config get-context          #(To view all the contexts)

kubectl use-context CONTEXT-NAME    #(To use the context)

This will let the user to access the K8s resources as much given in role.yaml file through kubectl. But to make RESTapi-call the api-server without kubectl command we can curl using

curl -k https://API-URL:64418/api/v1/namespaces/default/pods --key USER-PRIVATE-KEY.key
--cacert CA.cert --cert USER-NAME.crt

Interacting with the K8s cluster resources can be done with two types of users.

  1. Human users

For them, we have to create non-admin users and assign roles to them.

  1. Service Accounts.

But when the cluster resources need to be accessed by the non-human users, like pods and jobs, and also sometimes when a webhook triggers something or monitoring resources need to see the other pods.

In these cases, they also need to verify to the API server as a user, and we create service accounts for them as well as bind certain roles to them.

In this example, we are going to create a service account, a long-lived API token secret, a role, and a role binding.

Previous Kubernetes versions used to automatically create a secret, then generate a long-lived api-token, then attach that secret to the ServiceAccount and mount that token inside pods using that ServiceAccount.

But now in the newer versions, it needs to be created manually by creating a secret, and Kubernetes will populate that secret with a token. Otherwise, now by default, Kubernetes uses Bound Service Account Tokens (Projected Tokens). These tokens are short-lived, automatically rotated, mounted directly into the Pod, and not stored as Secrets. As these are now also scoped to a specific Pod, this became more secure.

  1. Creating Service Account
kubectl create sa SERVICE_ACCOUNT_NAME
  1. Creating the secret
apiVersion: v1
kind: Secret
metadata:
  name: my-sa-token
  annotations:
    kubernetes.io/service-account.name: my-sa
type: kubernetes.io/service-account-token

save and apply this file.

  1. Creating Role to get pods
kubectl create role ROLE_NAME --verb=list,get,watch --resource=pod
  1. Creating the RoleBinding
kubectl create rolebinding ROLE_BINDING_NAME --role=ROLE_NAME --user=SERVICE_ACCOUNT_NAME

Now if we execute

kubectl get pods --as SERVICE_ACCOUNT_NAME

we can see the pods as a service account.