Kubernetes pi cluster
Building a kubernetes cluster can bring many benefits such as home automation, DIY project and testing purposes. This guide will provide guidance how to setup and connect your pi’s to communicate over the local network, followed by the installation of kubernetes thereby offering a wide range of options for projects. Additionally this guide provides a guidance for even an unexperienced Linux user.
Additionally to this post, I have created a script that automatis the steps found in this post, it can be found in the post Kubernetes on pi script I highly recommend to read this once finished with this post, to understand what the script.
In order to carry out this guide you must have access to the minimum requirements listed below.
- 2 Raspberry pi 4
- 2 SD Cards
- 3 Ethernet Cables
- 1 Ethernet switch (your home router might do)
- 2 USB Charger (5V2A-3A ea. pi.)
- 1 SD Card Reader
Prepare the Pi’s
Before building the pi’s the SD Cards must be prepared so that the pi’s can start and be configured. The following section will explain how to flash the SD Cards and prepare them for the Pi’s followed by a guidance of how to setup the Pi’s so they can boot and have access to each other over the network.
This will be followed by the two configurations sections one explaining the configuration of the Pi’s so they can be connected and configured remotely and the other the configuration and setup of kubernetes and how to access and control it.
Flashing/Preparing the SD Cards
Before starting there are a few different software are needed in order to carry out this part of the tutorial, the first program is Etcher which will allowing flashing raspberry Pi OS lite (Pi OS) on to the SD cards for the pi’s. I have used a specific version of Pi OS which works with this tutorial. I’ll recommend using this version or try with the latest one however there was some issues running on later version due to some differences in the configuration of the OS, where kubernetes had some issues with missing cgroup’s configuration which prevented kubernetes from running.
Once Etcher is installed and pi OS is downloaded, plug-in the SD card into SD card reader and open Etcher. In Etcher select the raspberry pi OS and select the SD card you wanna flash it on to and press flash. Once it’s done flashing the SD card go to the SD card boot partition and create two files one called “ssh” which should be left as a blank file, another file should be called “cmdline.txt” which must include the following line “cgroup_memory=1 cgroup_enable=memory”. Now the SD card is ready to be inserted into one of the pi’s. Follow the procedure for each of the SD cards.
The file named “ssh” will be read by Pi OS on boot and thereby setup a SSH service to connect to. The file named “cmdline.txt” is used for kubernetes and is also read when the Pi boots, allowing the cgroups to be enabled for kubernetes.
Prepare the Hardware
Now that the SD cards are ready after following the section above the pi’s can be connected to one another. Following the steps below should enable the pi’s to connect to your local network and become discoverable for the next step where they will be configured. I have Attached an image of my cluster showing how it can be connect with it’s own switch. Please remember using a seperate router instead of a switch might cause you problems as each router defines IP’s addresses so if your computer isn’t connected to the same router as the pi’s you might not be able to carry out the next section.
- Insert each of the SD cards to each of the Pi’s.
- Connect one end of the Ethernet cables to the LAN port of each of the pi’s
- Connect the other end of the Ethernet cable to the Ethernet switch
- If using a separate switch specifically for your cluster, connect the last Ethernet cable to your home router thereby connecting all the pi’s to the internet
- Power on the Pi’s and verify that the pi’s light up and is powered.
The image above is the setup I use. The two Pi’s connected to a switch (bottom right unit) using the white Ethernet cables, and connected to power via the USB power supply (bottom left unit) on USB 1 and USB 2. The Ethernet switch is powered via USB on USB 3. and connect’s to the internet using the yellow Ethernet cable going to the local network router.
Configure the Pi’s
Now that the pi’s are running Pi OS, they should be discoverable on the network thanks to the ssh file. In order to carry out this step we again need some software that can discover our pi’s on the network, I’ll recommend IP Range – Angry IP Scanner and this will be used for just discovering our pi’s before they have a static local IP. (Note Windows users might need to download the JDK found here https://adoptopenjdk.net/). Please download and install the software before proceeding.
Once Angry IP Scanner is installed start the program, and run the scanning by pressing the start button, this should find all devices on the network, however the pi’s should appear with the hostname “raspberrypi”. Remember the IP addresses these will be used in the next section to access the Pi’s using SSH.
Accessing the Pi’s
This step requires another tool in order to access the Pi’s using SSH there are many ways to do this however this guide uses the program Termius as it offers easy profiling and configuration that is easy to get started with. Before continuing please download and install Termius.
Now let’s configure Termius so it’s easy to access the Pi’s onwards, first add a new Host, and enter give the host a name and enter the IP for each of the pi’s remember to give the pi’s different name to distinguish the two from each other. I named mine “PiNode0” and “PiNode1” and both assigned with the tag named “piCluster”. Finally, add the username and the password which by default on Pi OS is password: “raspberry” and username: “pi”
Once finished you should see your pi’s in the menu point Hosts, now double click on the pi in order to access it, once accessed it will ask you for saving an RSA fingerprint, chose add and continue so that the handshake between your device and the pi is stored as a known host. Finally, a terminal is shown which is the terminal of the pi, commands executed here is carried out on the pi.
Configure and secure your Pi’s
This step will carry out a little additional configuration such as changing the hostname of the pi’s for easier to remember them and adding a more secure access to the pi’s by changing the SSH password or adding a key pair that must be defined in order to access the pi’s in the future.
Changing the hostname
Let’s start by changing the hostname of the Pi’s, thereby making them even more distinguishable from one another by entering the following command.
sudo nano /etc/hosts
The “sudo nano” part of the commando will open an editor enabling you to set a new hostname for your pi. The editor can be saved/closed by pressing CTRL + X followed by Y for saving and N for closing without saving.
The first commando will show the pi’s hostname first it might be written “127.0.1.1 raspberypi.local”, change this to piNode0 and press CTRL + X followed by Y. Change it so that it matches the hostname given in Termius.
Next do the same for the hostname file, entering the following command and modify the file so that it matches the image below. Lastly, restart your pi with the last command in this section.
sudo nano /etc/hostname
When the Pi is restarted let’s see what we did, Start Angry IP scanner and run it, and see that the pi’s hostnames have changed to what we just defined.
Now open Termius again and let’s configure the hosts to match the new hostnames, right click on each of the hosts and press edit and change the IP addresses to match the hostnames found in Angry IP Scanner.
Now try to access the pi’s once again, and verify that the hosts are still accessible, otherwise, try to backtrack and see if a step wasn’t carried out correctly.
Changing the default password
Let’s change the pi’s passwords, access each of the pi’s using Termius and enter the two commands below, the first will ask you to define a new password the other will restart the pi in order for the new password to be used. Remember the password; you’ll need to change this on Termius afterward in order to be able to access the Pi’s again.
Creating key pairs
This section is optional, however highly recommended to add a bit more security to your cluster by using a key pair to access the Pi’s over SSH.
This guide differs a little depending on if you are on a Windows or Linux/Mac device, feel free to follow the steps that match your system.
first check if you device has a .ssh folder, by entering the command below, if you get “No such file or directory” skip forward to the generate key pairs section below.
Windows: cd %userprofile%/.ssh Linux/Mac OS: cd ~/.ssh
in case this folder does exist verify that the folder is empty by entering the command below, if the folder is empty skip forward to generating key pairs.
Windows: dir id_* Linux/Mac OS: ls id_*
Otherwise backup the files found in the folder by entering the following two commands.
Windows: mkdir key_backup Windows: copy id_rsa* key_backup Linux/Mac OS: mkdir key_backup Linux/Mac OS: cp id_rsa* key_backup
Now the .ssh folder content should be found in the key_backup folder found within the .ssh folder, thereby preventing unnecessary overwrites when the key pairs will be generated in the next step.
Generating Key Pairs
Now that we have prepared for the generation of the keys make sure to open your command prompt or Terminal depending on your system, Windows user must open it as an administrator.
Enter the following command and follow the steps asked in order to generate the key, you will be asked for a keyphrase, make sure it is unique and must be remembered as it will be needed later.
Once the key-pair has been generated go to your .ssh folder and verify your new key-pairs are generated and go ahead and rename these so you can remember what these keys are for, I have renamed mine to the same name as the pi’s so they can be distinguished from one another. Go ahead and generate a key-pair for each pi in your system, by repeating this section (Generate Key Pairs).
Applying Key Pairs
To carry out this step, open Termius and access each of the pi’s carrying out each of these steps thereby applying your public keys to the raspberry Pi’s.
Open up Termius and access each of your pi’s and enter the following commands below.
mkdir .ssh cd .ssh nano authorized_keys
The last command creates a file named “authorized_keys” this must contain the content of the public key, generated in the previous section (piNode0.pub). Copy the content from the piNodeX.pub and insert it into Termius and save the file (authorized_keys).
Once the public key is on the Pi enter the following commands, thereby setting the Owner’s rights to be read and write access to the file, which ensures the file isn’t unnecessarily tampered with.
chmod 600 authorized_keys cd .. chmod 700 .shh
Now that, that’s configured let’s ensure that we can use these keys to access the Pi’s using Termius, go to the keychain, and add a key for each of your key pairs.
Entering the passphrase used when you generated the key pairs will be used here as well as the private key, adding the public key for good measure and Termius now have all the information it needs to access the pi’s using the private key and acts as a backup for the public key as well.
Finally, use the key by editing the hosts, and set the keys we just configured in the steps above, now Termius will access the pi’s using the private key and have a secure connection when communicating the Pi’s.
Disable password authentication
In order for the key pairs to be used at all times we must disable normal password authentication on the Pi’s resulting in the key-pairs must be used in order to communicate with each Pi.
Enter the following command on each of the Pi’s and modify the configuration so that password authentication is disabled.
sudo nano /etc/ssh/sshd_config
change the configuration that states #PasswordAuthentication yes to PasswordAuthentication no, save the file and reboot the Pi.
That concludes the secure SSH connection to the Pi’s, next we’ll configure the static IP’s which will help when you need to remember and access the Pi’s without reconfiguring or finding the Pi’s if something on the network should change.
Configure Local static IP
Configuring a local static IP is good if you don’t wanna bother with the IP changes when you change anything on your home network or if the Pi’s suddenly get assigned a new IP. This way the Pi’s request a static IP which is easier to remember when developing or connecting towards the cluster.
The first step is to access the pi’s using Termius and enter the command below, resulting in a line containing the pi’s default gateway.
ip r | grep default
In the next step enter the command below, resulting in seeing the “resolv.conf” which contains the IP addresses for the domain servers, remember these, and close the file.
sudo nano /etc/resolv.conf
Enter the following line will provide you with the correct file where we need the IP’s we just obtained.
sudo nano /etc/dhcpcd.conf
Add the following lines to the bottom of the file, fill in the IP addresses for your Pi, feel free to backtrack and add the IP addresses we just found.
interface eth0 static ip_address=*REQUEST_IP*/24 static routers=*DEFAULT_GATEWAY* static domain_name_servers=*DOMAIN_SERVER*
Save the file and reboot the Pi by entering the command below. Once the Pi is restarted feel free to use the Angry IP Scanner and verify that the IP for the Pi has changed.
Now the pi’s are ready for us to install kubernetes. In such a cluster there are two different units, one that orchestrates the cluster known as the Master and the ones following the Master pi is the workers. Installing Kubernetes is required to be carried out on all the Pi’s thereafter one of the pi’s must be configured as a Master and another as a worker.
Accessing your pi’s and enter the following commands in order to install kubernetes on each of the pi’s. Before we start know that if you ever want to uninstall kubernetes your can do so by using one of the commands below.
sudo /usr/local/bin/k3s-agent-uninstall.sh sudo /usr/local/bin/k3s-uninstall.sh
Your can also remove nodes from your cluster by entering the command lines found below.
sudo kubectl get nodes sudo kubectl drain <node-name> sudo kubectl delete node <node-name> sudo reboot
Setup Master Node
On one of the nodes you must enter the following command, this will require your pi has internet access as it will download the lightweight version of kubernetes known as k3s, and install it.
curl -sfL https://get.k3s.io | sh -
Use the following command to see your nodes, for now it should return only the master node, but once the workers has been configured you can run this command again and verify that the nodes are connected to the cluster network.
sudo kubectl get nodes
Once it has downloaded and installed enter the following command, which will provide you with a token you must save for the configuration of the worker nodes, as it is a connection token the pi’s uses to make a handshake, this token will be referred to as K3SToken.
sudo cat /var/lib/rancher/k3s/server/node-token
The token should look similar to the code found down below.
Once your workers are connected access you master node and assign each of your worker nodes with a worker role in order to distinguish the pi and their roles within the cluster nodes.
kubectl label node <NAME_OF_NODE> node-role.kubernetes.io/worker=worker
Setup Worker Node
To setup the worker node is straight up entering a sequence of command lines on the pi, what you’ll need is the master node IP and the K3SToken which was introduces in the Setup Master Node.
curl -sfL http://get.k3s.io | K3S_URL=https://xxx.xxx.xx:6443 \ K3S_TOKEN=<K3SToken sh -
For a more accurate example see the following command lines.
curl -sfL http://get.k3s.io | K3S_URL=https://xxx.xxx.xx.:6443 \ K3S_TOKEN=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA::server:BBBBBBBBBBBBBBBBBBBBBBBBBBBBB sh -
Now the worker should have k3s and made a handshake with the master node found using the Ip. Go back to the Setup Master Node to assign the worker with the worker role.