Every Kubernetes tutorial starts the same way: spin up Minikube, run kubectl get pods, see a green checkmark, feel accomplished. You've learned the CLI. You haven't learned Kubernetes.On a single node, scheduling is trivial: there's only one place to put things. Networking between nodes doesn't exist because there are no other nodes. Failover is impossible. You're practicing orchestration without an orchestra.I wanted to see what happens when the scheduler has to make actual decisions. So I connected the two machines on my desk: a Ryzen 5 5600X desktop running Ubuntu 24.04 and a MacBook Pro M4 Pro.Different operating systems. Different CPU architectures. Same cluster.
Why k3s, Not kubeadm
Standard Kubernetes installation (kubeadm) requires managing etcd separately, configuring multiple system services, handling certificate rotation, and budgeting at least 2GB RAM for the control plane alone. k3s bundles all of this into a single binary under 100MB. Same Kubernetes API, same kubectl, same YAML manifests. The difference is operational overhead, not capability.For learning, that trade-off is obvious. Spend time understanding scheduling decisions and network topology, not debugging installation issues.One curl | sh command, and the control plane was running.
The Part Every Tutorial Gets Wrong
Here's where it gets interesting. k3s doesn't run on macOS natively. The standard solution is Multipass (Canonical's lightweight VM tool), which spins up an Ubuntu VM on the Mac in seconds.The gotcha: Multipass creates VMs on an internal NAT network by default. The Mac can reach the VM. The Ubuntu desktop cannot. And Kubernetes requires bidirectional connectivity between all nodes.The fix is bridged networking:
That --network en0 flag gives the VM a second interface on the home network's subnet. Now both machines see each other directly.But there's a second gotcha, and this one is subtler. Even with bridged networking, k3s and its overlay network (flannel) will default to the NAT interface for VXLAN tunnels. The node appears Ready in kubectl get nodes. Everything looks fine. Then pods on one node can't reach pods on the other, and you spend an hour staring at flannel logs.The fix: explicitly tell k3s which interface to use.
Two flags. Sixty minutes of debugging if you miss them.
The Moment
NAME STATUS ROLES AGE VERSION ARCH
master-fluffy-ms-7c94 Ready control-plane 41m v1.34.5+k3s1 amd64
k3s-worker Ready <none> 98s v1.34.5+k3s1 arm64
Two nodes. Two architectures. One cluster.When I deployed nginx with four replicas, the scheduler distributed them: two on the Ubuntu PC, two on the Mac VM. No special configuration. The nginx image is multi-arch, so each node pulled the version built for its CPU automatically.I ran a curl pod on one node that hit an nginx pod on the other node by name (http://nginx). Cross-node, cross-architecture, transparent. This is the thing single-node setups can't show you.
What Draining Teaches You
The most instructive moment wasn't deploying. It was draining.
This marks the Mac node as unschedulable and evicts all running pods. Kubernetes reschedules them to the Ubuntu PC. You watch containers disappear from one machine and appear on another in real time.Then kubectl uncordon k3s-worker brings it back. New pods can be placed there again. This is the operational reality of Kubernetes that no amount of single-node practice can simulate: graceful maintenance, workload migration, capacity management.
Architecture-Aware Scheduling
Once you have two architectures, you can constrain scheduling with nodeSelector:
nodeSelector:
kubernetes.io/arch: amd64
Hard constraint: this pod only runs on x86 nodes. Or use node affinity for soft preferences: "prefer arm64, but amd64 is fine if arm64 is full."This matters in production. Edge deployments mix architectures. Cloud providers offer ARM instances at lower cost. If your cluster speaks both dialects, you have options.
The Numbers
Total cost beyond existing hardware: $0
Time to working cluster: ~2 hours (including the flannel debugging)
k3s binary size: under 100MB
Control plane RAM overhead: ~512MB (vs. 2GB+ for kubeadm)
Time to deploy and verify cross-node workloads: 10 minutes
What I'd Do Differently
Nothing, honestly. The flannel networking bug was the best teacher. If it had worked on the first try, I'd understand less about how overlay networks function. The error was the lesson.If you have two machines on the same network, you have everything you need for a real Kubernetes cluster. The concepts that matter (scheduling, networking, failover, maintenance) only emerge when there's more than one node. A home lab with two machines teaches more than a month of managed cloud K8s tutorials.The full setup guide, with every command, flag explanation, and troubleshooting step, is on the project page.
