How do you really know how it works if you do not work it? Serverless? Clusters? Linux? The most skilled people in technology are well practiced. What if you want to push yourself to the next level? Maybe you want to experiment with some things that are a little sketchy and you dont want it escaping to the Internet. Maybe you want to test that security thing you saw on that thing that time.
The answer is to setup a lab. Something that can have a network and services but be completely disconnected from your home network and especially be disconnected and air gaped from the Internet.
There are several routs one can go to set up a lab. To have several machines they can be virtual or hardware. I did not plan for this project so the hardware budget is non-excitant. I don’t have any unused hardware laying around that can run VMs. So lets setup a CLUSTER OF RASPBERRY PIS! enter…
Cluster Hat SugarPi
… OK, yes, there are plenty of tutorials out there on how to do this, but mine is somewhat special because it is using 8086’s ClusterHat to host four Raspberry Pi Zeros in USB Gadget mode on a Raspberry Pi 3B, powered by a PiSugar2 Pro. In theory it could serve up a WiFi access point to a little portable network. Although, I am not sure that would run very long, but it is still cool to be able to have the option to just plug it into it.
Configuring the ClusterHat from scratch isn’t overly complex, but I am going to go ahead and use the images furnished as a starting point. The basic setup will give us a sequestered network that can be airgaped. When plugged in, it is NATed so we can do updates and downloads. On top of this private network of tiny computers we will install Docker Swarm for some cluster fun. This post is organized into three parts:
To build my cluster, I started with a Raspberry Pi 3B+. (if you decide to use a Raspberry Pi 4B consider the Pimoroni Fan Shim to keep it cool) To it’s base I added a PiSugar 2 Pro for portable power. This highlights the lower power consumption of the Raspberry Pi 3 compared to the 4 as well. Then I followed the instructions to install a Cluster Hat. I was happy to find out that these two pieces of hardware do not use conflicting I2C addresses. To the Cluster Hat I plugged in four Raspberry Pi Zero W (note leaving the headers off make for a better fit). Through out this tutorial I will refer to these Raspberry Pi Zeros as zero or p or Zero P Hosts. In the examples I will often refer to the #2 Zero P Host in the examples to make sure you can see that it is a number and not the letter “I” since they tend to look the same and there is plenty of ‘pi’ references as it is the default user. To top it all off I added 5 Samsung 32GB SD cards and ran the USB cable from the Pi3B USB A ports, under the hat to the hat’s power supply.
I followed the Intermediate instructions on the Cluster Hat site using the standard method to flash the CNAT full image on the SD card for the controller. Then I use the P1-4 lite image on the Pi Zero SD cards. Note that you can boot them without SD cards, but we will keep it simple for now. I also touched the ssh file to enable it on each of the SD cards.
On the P1-4 images, open the command.txt and add the following after
ipv6.disable=1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
Note that it is not required to disable IPv6 but if you are using NAT like I am it is ok.
You can also open config.txt and uncomment the last line or add the following to enable gadget mode:
If you started from scratch with Raspberry Pi OS you may also need to add the following to the end of the line in cmdline.txt:
Next I put in the SD cards to their respective places and plugged in an ethernet cable to the controller Pi (the 3B). Then I connected via ssh and completed the standard Pi setup on the controller, being sure to change the pi user password. You can use
sudo raspi-config or
passwd or however you are comfortable.
I then ran
clusterctrl on then
clusterctrl off a couple times just for the squee factor of watching the LEDS chain on. With the cluster on, I did a quick
lsusb -l to make sure they are attached and look at
ifconfig to see they are connected to the host.
Lets do some DNS. On the controller Execute
sudo nano /etc/hosts and add the following to the end of the file:
Then we will want to copy our ssh key to each zero to make setup easy:
ssh-copy-id -i .ssh/id_rsa.pub p1
ssh-copy-id -i .ssh/id_rsa.pub p2
ssh-copy-id -i .ssh/id_rsa.pub p3
ssh-copy-id -i .ssh/id_rsa.pub p4
Docker on Controller
While on the cluster controller install docker. You should note that the convince script does use apt-get so you don’t have unmanaged binaries lying around.
curl -sSL get.docker.com | sh
Then you can initialize the Docker Swarm:
sudo docker swarm init --advertise-addr 172.19.181.254
In the output from
init there is a line that starts with ‘
docker swarm join‘. That is your command to join the swarm, you might want to copy it to your notes so you can copy and paste later.
Now issue the following command to see that the Status is ‘Ready’ and Availability is ‘Active’. Also notice that Manager Status is ‘Leader’.
sudo docker node ls
The the first Pi Zero or P1 (Pee-one) will be a manager, to get that join command we ask docker for it with:
sudo docker swarm join-token manager
Copy that out and save it in your notes.
Connecting to the zeros is as easy as
ssh pi@p2. Alternatively, if you want to setup over serial you can
sudo minicom p2 since aliases are nicely setup for you if you use the standard p1-4 images. It may not be pretty over serial, but it gets the job done.
After finishing the configuration and rebooting each Pi Zero I want to make sure to get the system up to date by typing:
sudo apt-get update && sudo apt-get -y upgrade
Once this is done on each zero, you will want to disable swap.
sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove
On each zero you will want to add a hosts entry for the controller, generate a ssh key and copy it back to the controller, do the following for each zero (p1-4):
echo "172.19.181.254 cnat" | sudo tee -a /etc/hosts && ssh-keygen && ssh-copy-id -i .ssh/id_rsa.pub cnat
This will generate a ssh key and copy it back to the controller after asking you some stuff, like login.
From the now live Pi Zero Hosts
Now lets do the docker thing to the Zeros using the convenience command. On P1 (Pee-one) use the docker join command you copied in the last section to join it as a manager. You will see “This node joined a swam as a manager.” You can issue a docker node list to see the two nodes.
Now on P2-4 use the first join command from docker init to join them to the swarm as workers. After you issue the command it should output “This node joined a swarm as a worker.” (notice the Docker team apparently likes their punctuation.) When you get done list the nodes, and you should see the following:
Then lets add a nice tool to visualize what is going on. On the Cluster Controller issue the following:
sudo docker service create \
--name viz \
--publish 8080:8080/tcp \
--constraint node.role==manager \
Then you can see that it is running on one host by listing the docker services:
sudo docker service list
Now you can hit the public cluster network address on port 8080. I registered mine in my local DNS under the name cluster-hat so I get there with
So now, for a quick demonstration. While watching the vis service, install a ping service and scale it up. To install the service use:
sudo docker service create --replicas 5 --name party alpine ping myhacker.party
When you hit enter it will start 5 services pinging the
myhacker.party domain. You may see them bounce between the different servers.
Then scale it down:
sudo docker service scale party=1
Then remove the service with:
sudo docker service rm party
The ClusterHat is very cool. After I backup images of everything, I will have a great restorable lab environment. Setting up Docker is easy enough and seems to be the only well documented clustering solution that scales down to a Pi Zero. Tell me if you know of something better. With the large selection of tooling and runnable apps, this seems like a fun way to experiment.
So why did I not use Kubernetes? If you were to count words on the internet Kubernetes is only rivaled by the word ‘the’. However support for the particular processor on the Raspberry Pi Zero was dropped when an compiler switch was introduced to compile Kubernetes more securly. According to GitHub the last version to work on the Zero was version 1.5.6. So to run any newer version meant a custom kernel and a custom compile of Kubernetes itself. So k8s will be something I will land on my RPi 4 Cluster or my Cluster Triple (coming soon).
Azure this is not, but I can unplug it from all networks and play with networked things without leaking onto the Internet. With the WIFI modules I could use this battery powered cluster as a multi-headed pwnagotchi. The fun and learning is endless.