Some backstory of container workloads

Before talking about K8s, let's discuss some history of containers.

In the early days of Linux there was a need to isolate the processes so that they could work independently without bothering other processes. Suppose process A needs a separate PID, network, mount points, etc., and process B also needs a similar setup; to counter this situation NameSpaces came into picture. The namespace provided each process their separate isolated environment as they needed. Like this,

Linux Machine
│
├── Namespace A
│   ├── Processes
│   ├── Network interfaces
│   ├── Mounts
│   └── Hostname
│
└── Namespace B
    ├── Processes
    ├── Network interfaces
    ├── Mounts
    └── Hostname

Later this became the base and fundamental of containers.

Now another problem came of utilizing the resources by processes. Suppose a group of processes (not namespaced) goes wild and starts consuming the entire memory and CPU; in this case developers came up with a solution to limiting the resources to a particular group of resources. This idea is called Control Groups or cgroups. Suppose a web application group of processes is only allowed to consume 2 CPUs and 500 GiB of RAM.

You can create a namespace and Cgroups with the terminal; check these commands out.

sudo unshare --pid --fork bash

sudo ip netns add mynet

These two give you a new PID and a network namespace. And for cgroups try

sudo mkdir /sys/fs/cgroup/mygroup #creating cgroup named mygroup

echo $((100  1024  1024)) | sudo tee /sys/fs/cgroup/mygroup/memory.max #limiting memory to 100MiB

echo "50000 100000" | sudo tee /sys/fs/cgroup/mygroup/cpu.max #limiting cpu to 50%

echo <PID> | sudo tee /sys/fs/cgroup/mygroup/cgroup.procs #with this you can put a process with PID in the mygroup

If you give the PID of the namespace to the newly created cgroup, it will become like a primitive container-like setup, but you also need Filesystem, Networking and Security features as well. But with this you have created a basic container-like workload.

Combining all this gives us isolation, resource control, storage, networking, and a runtime to manage these little compute boxes, which becomes the idea of modern containers.

How modern containers run

We talked about a runtime, right? This will manage all the above requirements and provision the workloads. There are multiple runtimes like crun (written in C), youki(written in Rust), kata containers, gvisor, but Open Container Initiative (OCI), which standardized and created a runtime to work with the Containerd, Docker, and K8s, called Runc, which completes all the OS requirements (namespace, cgroups, networking, etc.) and creates the containers.

This runc is the smallest level manager of the containers, but for running multiple containers, we need a higher-level container manager for work like container lifecycle management, dealing with container metadata, and recovery after failures. But instead of directly integrating this higher-level container manager to runc, we use another service in between to counter if the higher-level manager fails or gets restarted. Then all the running containers will die. For that a small process named shim is introduced. This Shim runs the containers and manages Runc even if the higher-level manager restarts.

On top of Shim, the higher-level container manager runs and manages jobs like dealing with the container images. 3 runtime managers are the most popular in the container world, Podman and Containerd. CRI-O is more Kubernetes-focused, and containerd is more general-purpose. Podman is daemonless, and its architecture is different than these two. Podman and CRI-O both use conmon (container-monitor) instead of Shim. And Containerd uses Shim. Containerd also works with both Kubernetes and Docker Engine.

With this you got the idea of managing and creating the modern containers. Now we are going to interact with them.

Interacting with containers

For interacting with or using the containers, I am taking two scenarios: first, using Docker and second, Kubernetes.

Docker Engine (dockerd) currently uses containerd underneath, and we can interact with the containers using docker-cli with docker commands. But for Kubernetes we need a Container Runtime Interface (CRI) to interact with the runtime manager. As Kubernetes previously used Docker-shim and docker engine as the container runtime manager and now decoupled using CRI so that it can use any container runtime, like CRI-O or Containerd. And eventually Docker also started using Containerd instead of its own docker-shim.

Now this Container Runtime Interface gives Kubernetes freedom to use multiple runtime managers, but to interact with it (runtime manager), we needed a tool. This is how cri-ctl comes into the picture. It is a command-line tool for interacting with the Kubernetes container runtime manager through the CRI. Crictl commands are similar to docker commands for interacting with the containers. Like crictl ps and docker ps to list the containers. So now we can directly interact with the containers running inside the pods (layer on top of containers). Think like containers are running under containerd; you as a user and K8s both can interact with them, K8s using kubelet (K8s service to create Pods) and you with crictl.

Let's make this brief and clear now. So OS Namespace and Cgroups create basic containers like workloads; runc creates this more efficiently; shim makes sure to run the containers in the absence of containerd; containerd manages all this and also ensures container lifecycle management; Kubelet interacts with containerd using CRI; as a user, you can also interact with the containers using Crictl.

Kubernetes components and architecture

Kuberentes is a platform consisting of multiple services coupled and running together with the sole purpose of managing the container workloads in the most efficient way possible. It consists of master nodes, which manage the workloads, and worker nodes where workloads run.

Before discussing how it does, let's get to know some components or services of it first. One we already came across, yes, the Kubelet. We can say Kubelet is the parent or top-level manager of all the other services.

It runs as a process on every node. You can check with

systemctl status kubelet

if it is running or not. Kubelet is the only component that runs as a process, and other components run as pods, and kubelet manages or creates them. First let's discuss all the other components. For now, think pod is an abstraction layer on top of containers, which is managed by kubelet.

One more thing: both kubelet and containerd run as an OS process. Kubelet asks containerd to manage the containers, and all the other K8s components are managed and created by kubelet (including pods). Let's know something more about Kubelet.

Kubelet

It runs as a process on every node, monitors every resource, starts and stops containers, checks resource and health information of the node, and manages mount volumes. It constantly compares the desired state and the current state of the cluster and makes sure to create the wanted resources or changes to the cluster via the following components. These resources run only on the master node, and all the workloads run on the worker nodes, but kubelet runs on every node. Kubelet creates the following resources as pods (specifically static pods).

API server

This is the first point of contact to the cluster. For every command you type through kubectl (Kubernetes CLI), every request comes to the cluster; whether the application request or resource modification request, it redirects it to the responsible component of that request. Although Kubelet creates it, it still requests to kubelet goes via this HTTP API server.

ETCD

Etcd is a distributed key-value database. It was a standalone database project before kuberntes. Kuberntes started using it as its database service. As the database suggests, it keeps all the data of the cluster, and the API server interacts with it. As it is a distributed system, we can create multiple database instances of the cluster and make it fault tolerant. If you are using Kubeadm for cluster setup, this also runs as a pod by kubelet. But you can deploy it as a standalone database server outside of the cluster as well.

Controler Manager

Previously told that kubelet takes the cluster to the desired state from the current state. This is done by the controller manager. It does the actual job. The controller manager runs various controllers that continuously compare the current and desired state. When we apply a manifest file (where we define what the things are we want in the cluster), the controller manager makes sure to fulfill the things written in the manifest file. Suppose we want 3 pods now; instead of 1 pod, it does that.

Scheduler

Let's take the above example. Now the controller manager makes sure to make the pods from 1 to 3, but where to create them or when to create them is memory available for them. These things are checked by the scheduler, and then the scheduler creates the pods.

Now let's summarize this a little bit: we want 3 pods of some application. We created a manifest file, through kubectl commmand we applied the manifest file. Now the request goes to the API server, and then it tells the kubelet. The kubelet informs the controller manager to create the following resource: the controller manager checks with etcd about the current and actual state of the cluster; if it is not in the desired state, the scheduler is informed to create the pods. Now the scheduler checks which node is suitable for the pod after checking resources and other constraints like taints (already written in which node to create) and then creates the actual pod there. Kubelet and Containerd are also running on the node; they do the next job of running the pod and the container.

Core DNS

Now comes the networking part. How every resource and component communicates with each other, obviously by IP addresses. This Core DNS is the internal DNS service of kubernetes for finding the resources. This also runs as a pod by kubelet.

Installation

Using a Kubeadm setup for now, although there are multiple other ways you can provision a cluster like Rancher, Minikube, K3s but here we are going to use Kubeadm, as it is all done manually, and you can actually see the things that we are going to install and run.