A step-by-step guide to setting up a Raspberry Pi Kubernetes lab with kubeadm, CNI, and CSI, including common pitfalls and maintenance tips.

Workflow: From Hardware to Cluster

1. Hardware Setup

   • Use Raspberry Pi 4 (4GB+ RAM) or Pi 5 for control plane and workers.
   • Ensure all Pis are on the same network with static IPs or DNS entries.
   • Flash Raspberry Pi OS Lite (64-bit) to microSD cards.

2. OS and Prerequisites

   # On each Pi:  
   sudo apt update && sudo apt upgrade -y  
   sudo apt install -y docker.io curl vim git  
   # Enable kernel modules  
   echo "cgroup_enable=cpuset cgroup_memory=1" | sudo tee -a /boot/cmdline.txt  
   

3. Install kubeadm

   curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -  
   echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list  
   sudo apt update && sudo apt install -y kubeadm  
   

4. Initialize Control Plane

   # On control plane Pi:  
   sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=<CONTROL_PLANE_IP>  
   # Follow prompts to set up kubeconfig  
   

5. Install CNI (Cilium Example)

   kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.14/install/k8s/cilium.yaml  
   # Wait for nodes to be Ready  
   kubectl get nodes  
   

6. Add CSI (Longhorn Example)

   kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.1/manifests/longhorn.yaml  
   # Monitor deployment  
   kubectl -n longhorn-system get pods  
   

7. Deploy Ingress (Traefik Example)

   kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.11/docs/content/getting-started/install-traefik-kubernetes.yaml  
   # Expose via NodePort or MetalLB (see tooling)  
   

Policy Example: Resource Quotas

apiVersion: v1  
kind: PersistentVolumeClaim  
metadata:  
  name: lab-pvc  
spec:  
  accessModes:  
    - ReadWriteOnce  
  resources:  
    requests:  
      storage: 5Gi  

Enforce quotas per namespace to prevent resource exhaustion:

kubectl create namespace lab  
kubectl create -f lab-quota.yaml -n lab  

Tooling

• kubeadm: Cluster bootstrapping (idempotent for repairs).
• Cilium: Lightweight CNI with network policy support.
• Longhorn: CSI for storage on Pi hardware (avoid etcd performance issues).
• MetalLB: Load balancer for external access (if using bare metal).
• Traefik: Ingress controller (lighter than NGINX for Pi).

Caveats

• RAM Constraints: Avoid Istio/Cilium together—Cilium alone can consume 1-2GB on Pi.
• Storage Overhead: CSI drivers like Longhorn require ~2GB per node for snapshots.
• Network Latency: Pi-to-Pi traffic may be slow; use wired Ethernet, not WiFi.

Troubleshooting

• Node Not Ready:

  kubectl describe node <node-name>  
  # Check kubelet status: systemctl status kubelet  
  

• CNI Not Installing:

  kubectl get pods -n kube-system  
  # Look for CNI pods in CrashLoopBackoff  
  

• DNS Issues:

  kubectl logs -n kube-system <coredns-pod>  
  # Ensure /etc/resolv.conf isn’t overwritten by OS updates  
  

Final Notes

Start small: master + 1 worker. Expand only after validating core components (CNI, CSI, ingress). Use kubectl top node to monitor resource usage and avoid overcommitting Pi hardware.

Source thread: How would you set this lab up?