Setting up multinode K8s Cluster

Overview

Installation of Kubernetes can be done through varoius ways. You can either choose to complete manual installation of every componets often known as K8s the hard way, or simply installing minikube. There are also other ways like Rancher and K3s but we are going to use Kubeadm here. Why kubeadm, because it gives us the opportunity to interact with each of the component after taking care of the repetative manual tasks itself. It is not harder like the whole manual installation but still gives opportunity of learning and seeing the things much internally.

We need to implement broadly the following steps after provisioning the similar nodes. I am taking debian as the OS of the machines. Kubeadm needs atleast 2 CPUs and minimum of 4GiB RAM per node to work smoothly. Although a cluster can have multiple master and worker nodes but all the initial configurations will be similer.

Kubeadm

Kubeadm itself does'nt runs kubernetes. It does some of the inital works before kubelet runs K8s components. It was created by Kuberentes community and released in Decemeber 2018 after Kubernetes 1.13 version.

After running kubeadm commands it checks for the kubelet installation, availability of ports, swap configuration, kernal and network settings. Then it creates kuberrnetes PKI(certificates for components), required static pods manifest files, kubeconfig file, etcd and api-server configuration files. You can check them on

ls /etc/kubernetes/manifests
ls /etc/kubernetes/pki/

in this directories.

Discussion on the steps

We need to follow the below steps sequentially in all the nodes.

  1. Configuring OS by tuning the kernel parameters to enable packet routing and disabling the swap.
  2. Installing the Containerd and configuring it for cgroups.
  3. Installaing Kubeadm, Kubelet and Kubectl.

Then on master node

  1. Configuring it as master node.

On worker node

  1. Joing them to create the cluster.

At last

  1. Implement networking using a CNI plugin for the whole cluster.

Lets explain some of the steps.

  1. We are enabling packet routing so that the network packets can reach to the pods, otherwise packets will go to the OS and it will reject them as packets are not distined for OS. Swap is very much slower and it cannot handel the pods load, so when memory is exhausted pods will start to use swap and the whole cluster will behave unwantedly, thats why disabling swap is important.

  2. Configuring the cgroups are one of the most important steps. Linux provides the cgroups which controls the resource allocation of the containers. Now to work with the cgroups we have cgroup drivers. This kubelet and Containerd can use different drivers, in that case the same container resources can be managed by two seprate drivers either cgroupfs or systemd. This creates confusion for the resources, so we are configuring containerd to use systemd driver as kubelet also uses it, otherwise containerd was using cgroupfs.

  3. Kubeadm can join the nodes together but container/pod networking it does'nt provide. So we need a seperate CNI plugin for that.

Time to open terminal

I am asuming you have already provisioned/created the number of nodes you wanted. Now its time to execute this steps on each nodes. First we will create 1 master node and join 1 worker node with it, and later you can join any number of nodes as another control plane or worker node with the same commands(little different for master node).

1. Configuring OS.

  • Changing kernal parameters for packet routing

execute this commands for both ipv4 and ipv6

sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1

Check using,

sysctl net.ipv4.ip_forward
sysctl net.ipv6.conf.all.forwarding

execute if sysctl command is not available

echo 'export PATH="$PATH:/usr/sbin:/sbin"' >> ~/.bashrc
source ~/.bashrc

or you can directly write to the sysctl.conf file

vi /etc/sysctl.conf

uncomment this lines or add this at the end

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

View using,

sysctl -p

another way to start this ip forward immeditely and then making the configuration persistent,

sudo sysctl -w net.ipv4.ip_forward=1

cat /proc/sys/net/ipv4/ip_forward

sudo tee /etc/sysctl.d/99-kubernetes.conf > /dev/null <<'EOF'
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system
  • Disabling swap

Open the fstab file and and comment swap

vi /etc/fstab

or directly disabling from cmd and view using free command.

swapoff -a
free -m

Now that we have configured the OS successfully, we will next go the containerd installation.

2. Installing Containerd and configuring it

  • Installation

First lets fresh up the repos

apt update
apt upgrade -y

Clean containerd if previously installed

sudo apt purge -y containerd
sudo rm -rf /etc/containerd /var/lib/containerd

Add the Docker repo first (debian),

sudo apt install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo $VERSION_CODENAME) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Now let's update the repolist and install containerd,

sudo apt update
sudo apt install containerd.io
  • Configuration

With this we are going to create containerd configuration file directory as with instllation it does not creates by default. Then we are outputing the containerd's deafult config file and writting a to file named config.tom in the directory.

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Now we are fixing the cgroup driver by setting SystemdCgroup to true from false to use systemd driver by default.

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

Or you can find "SystemdCgroup = false" section and make it "SystemdCgroup = true" in /etc/containerd/config.toml file.

Next we are going to restart containerd and verify using crictl

sudo systemctl daemon-reexec
sudo systemctl enable --now containerd
sudo systemctl restart containerd

Verify using

crictl info

The crictl command might not be installed with containerd, in that case install executing this. This will first specify the version, then downloads and unzip the file and installs it, then removes the zip.

VERSION="v1.36.0"
curl -LO "https://github.com/kubernetes-sigs/cri-tools/releases/download/${VERSION}/crictl-${VERSION}-linux-amd64.tar.gz"
sudo tar -C /usr/local/bin -xzf "crictl-${VERSION}-linux-amd64.tar.gz"
rm "crictl-${VERSION}-linux-amd64.tar.gz"

3. Installaing Kubeadm, Kubelet and Kubectl.

  • Making the repositories available
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
  • Installing them
sudo apt-get update
sudo apt-get install kubelet kubeadm kubectl

Now that we have kubelet installed we will enable it now and also pull the required images for faster completion.

sudo systemctl enable --now kubelet
kubeadm config images pull

Now till this we have to in all the nodes, Next we will setup the master node and create join token and apply that in the worker nodes.

4. Master node configuration

  • First we will create a configuration file and apply that with kubeadm init command.
vi kubeadm-config.yaml

add this

---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: <YOUR_NODE'S/MACHINES_IP>
  bindPort: 6443
nodeRegistration:
  kubeletExtraArgs:
    node-ip: <YOUR_NODE'S/MACHINES_IP>
kubeadm init --config=kubeadm-config.yaml
  • making kubectl commands available
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
  • Creating Kubeadm join token we will paste the output of the command in the worker nodes.
kubeadm token create --print-join-command

5. Worker nodes configuration

Either we can execute the output of the kubeadm token create --print-join-command on every node, or else we can join them using another config file.

create the config file

vi join_worker_config.yaml

add the following lines.

apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration
discovery:
  bootstrapToken:
    apiServerEndpoint: <YOUR_NODE'S/MACHINES_IP>:6443
    token: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    caCertHashes:
      - "sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
nodeRegistration:
  kubeletExtraArgs:
   node-ip: <YOUR_NODE'S/MACHINES_IP>

and execute this,

sudo kubeadm join --config join_worker_config.yaml

Now we have done all the essential configurations and join the nodes. Still we will do two things, Tainting the the master node so that application pods are scheduled only in worker nodes and implement the CNI.

  • Implementing the CNI, here using calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
  • To view and apply the taints on master node
kubectl describe node CONTROL_PLANE_NODE_NAME | grep Taints
kubectl taint node CONTROL_PLANE_NODE_NAME node-role.kubernetes.io/control-plane:NoSchedule-

That's it. We have now our Kubenretes cluster ready to work with. But before wrapping this up lets see some commands to play with the cluster and clearing the cluster configuration for a fresh start in case of any error occurs and also the guide to join multiple master nodes.

Joining multiple master nodes & reseting the nodes

For joining another master node we need a certificate key, we will generate that on current master node using,

sudo kubeadm init phase upload-certs --upload-certs

This gives a certificate copy that, and execute

kubeadm token create --print-join-command

and paste the certificate like this --certificate-key YOUR_CERTIFICATE_KEY at the output of the print-join-command on the second master node. Like,

sudo kubeadm join 192.168.1.10:6443 \
  --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <certificate-key>

this on the second master node. Noticed the difference with the worker node command? yes just the --certificate-key parameter at the end.

Now some more commands to play with the cluster and also reseting the cluster in case of any error.

  • Testing & Debugging
kubectl get nodes
kubectl get all -A
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get svc nginx
kubectl get pods -o wide
kubectl describe pod nginx
kubectl logs nginx
  • Cleaning up the cluster

Resting kubeadm

kubeadm reset
rm -rf /etc/cni/net.d
sudo rm -rf $HOME/.kube

Removing Kubernetes packages

apt-get purge -y kubeadm kubectl kubelet kubernetes-cni 
apt-get autoremove -y

Removing containerd

apt-get purge -y containerd

Removing Kubernetes and CNI directories

rm -rf /etc/kubernetes/
rm -rf /var/lib/etcd
rm -rf /var/lib/kubelet
rm -rf /etc/cni/
rm -rf /var/lib/cni/
rm -rf /opt/cni/

Cleaning iptables

iptables -F || true
iptables -t nat -F || true
iptables -t mangle -F || true
iptables -X || true

Cleaning nftables (if present)

nft flush ruleset || true

Deleting CNI interfaces

ip link delete cni0 || true
ip link delete flannel.1 || true

Removing CRI config

rm -f /etc/crictl.yaml

Final words

Thanks for tuning in, I hope with this guide now you have a complete Kubernetes setup. Exepriment all the different approaches and make sure to find more details about the steps we have done, this will give more inshight about the things how they are working under the hood. Bye, have a great day.