Foxmox: The Sleepy Proxmox Cluster

In the fall of 2003, I was learning Windows Server on a commuter train. Today, I have a miniature datacenter in my workspace. In my last post, I wrote about the small sparks that quietly shape a career. This is what thirty years of those sparks look like - not because everyone needs a Proxmox cluster, but because everyone benefits from having a place where it’s safe to learn.

Virtual Insanity

Virtualization is flat out awesome.

My first experience with it was Microsoft Virtual PC 2004. I honestly can’t remember whether it came from my MSDN subscription or arrived through Microsoft’s nonprofit donation program while I was working at NFTE, but I remember exactly what it enabled.

A complete Windows Server 2003 environment running on my laptop. That may not sound remarkable today, but at the time it felt like magic. I spent nearly two hours every weekday commuting on the Long Island Railroad. Instead of losing those hours, I had IIS, SQL Server, Active Directory, and a playground where I could learn ASP development and sharpen my Windows administration skills.

Those train rides taught me something I’ve carried ever since. Experimenting is a lot less intimidating when the machine isn’t your only machine.

Over the years virtualization quietly became part of almost every environment I touched. VMware ESXi. Xen. Hyper-V. VirtualBox. They all earned a place somewhere along the journey - then Docker arrived. For a while, containers replaced most of the virtual machines in my home lab - until Proxmox reminded me how valuable a good virtualization platform can be.

Meet Foxmox

This is Foxmox.

A 10-inch rack containing five Lenovo ThinkCentre mini PCs, an Alta Labs switch, a dedicated replication network, and far more capability than I originally intended to build.

Like many Homelab stories, it started with one perfectly reasonable idea: I wanted a single mini PC. Something inexpensive, quiet, and low power. A machine I could pull out when I wanted to experiment with a new operating system or application and tuck away when I was finished. Then I opened eBay. A seller had five ThinkCentres available as a lot and the plan escalated instantly.

What’s Inside

Foxmox isn’t impressive because of the hardware; it’s just a stack of mini-PCs that aren’t in cubicles anymore, fully depreciated and removed from a company’s IT inventory. It’s impressive because of what the hardware enables.

Three ThinkCentres form a small Proxmox cluster and a forth runs Proxmox Backup Server. The fifth – named Oberon – runs Ubuntu Server and hosts most of the services I rely on daily. They’re used business desktops – quiet, reliable, easy to repair, and cheap to upgrade.

The exact processor models and other specs aren’t particularly important; they get the job done. If anything, Foxmox is proof that you don’t need expensive hardware to build an environment capable of teaching enterprise concepts.

The Sleepy Cluster

“Why build a cluster if it’s turned off most of the time?”

Because it isn’t production infrastructure, it’s a workshop. My day-to-day services already live quite happily across Oberon, Chiron, Microab1, and my Synology. The cluster exists for client work, experimentation, and learning.

Need to evaluate a Linux distribution? Spin up a VM.

Need to build a proof of concept? Clone a template.

Need to intentionally break something so you understand how to recover it? Perfect.

When I’m helping an MSP evaluate an architecture or testing an idea before recommending it, Foxmox becomes the playground. When the work is finished, everything shuts back down. Idle compute is still compute, electricity costs money, memory wears, and fans collect dust. There’s no prize for leaving hardware powered on simply because you can.

Trust me; I ran a Cisco Catalyst 2900XL for about 60 months without a reboot to see what would happen – from day one in a datacenter until its closure. Nothing happened. The switch ran on an internal segment until the entire cabinet was powered down.

Why Proxmox?

One of the things I appreciate most about Proxmox is that it gets out of my way.

No licensing conversations, no artificial limitations, and no feeling like I’m fighting the platform. Just tools, snapshots, templates, clusters, Ceph and backups. The dedicated gigabit replication network lets cluster traffic stay isolated from everything else, and Ceph has been a fantastic excuse to learn distributed storage without needing enterprise hardware.

I don’t use these technologies because I need them every day. I use them because understanding them makes me better at solving problems for other people.

Backups on Backups on Backups

One of the ThinkCentres is dedicated entirely to Proxmox Backup Server, and while that may seem wasteful – after all, I could run it as a VM on one of the cluster nodes – it’s getting 8GB RAM from me one way or another. It’s better off on its own instead of allocating RAM that could be used by pretty much anything else on one of the nodes with more memory.

It’s also a critical component of the setup and deserves its own system. Foxmox00 has a 2TB SD for local backups with fast restores. Those restores are replicated to my Synology and another copy lives in cloud-based block storage.

I don’t treat 3-2-1 as doctrine in my own environment. I treat it as a baseline pattern for production systems, not a law of physics. In my Homelab, I’m not designing around SLAs, RTO, or RPO. I’m designing around how much failure I’m willing to personally absorb and recover from. That means multiple independent copies of data, retention strategies that assume mistakes will happen, and at least one geographically separated backup repository.

Best practices matter - but so does understanding when you’re operating outside the assumptions those practices were designed for. That said, blowing things up on the reg is sort of my thing, and learning how to recover is where some big lessons begin.

Oberon

If Foxmox is the workshop… Oberon is the workbench that’s always in use.

It wasn’t always that way. Before Oberon, my primary Docker host was a much larger machine affectionately named BigSexy. Yes, that was really its hostname. BigSexy mention at around 6:30.

It became memorable enough that I ended up talking about it during an appearance on the DNSFilter podcast, where I admitted that, despite the name, it had become hilariously overqualified for the job. A Xeon processor, 64GB of RAM, a dedicated GPU…all so it could spend most of its day running Docker containers that barely made it break a sweat. Eventually, practicality won.

I retired BigSexy, moved everything to Oberon, and never looked back.

The new machine consumes a fraction of the power, takes up almost no physical space, and quietly handles documentation, automation, source control, RSS aggregation, CRM, PDF services, and a growing collection of containers without complaint.

The GPU and RAM found a new home, and nothing went to waste. I appreciate powerful hardware; I just appreciate efficient hardware even more.

Ubuntu Earned My Trust

I’ve always leaned toward Debian and I think part of that comes from learning Linux through Slackware. Slackware teaches a certain sensibility – start with very little, understand every component, and add only what you need.

That train of thought naturally pushed me toward Debian for years and Ubuntu was never really on my radar. Ironically, Oberon runs Ubuntu because I had an unexpected urge to play Tertinet – a college LAN throwback – and the distro happened to make getting it incredibly easy. While multiplayer Tetris was fun, the nostalgia wore off after a couple of weeks and Ubuntu remained. Docker’s running perfectly and everything simply works.

Without realizing it, Ubuntu had earned my trust. Enough that, when it came time to rethink the operating system on my Dell laptop, I installed Ubuntu Desktop without much hesitation. Nobody convinced me to switch; months of smooth sailing using Ubuntu Server did that on its own.

Building Small Doesn’t Mean Thinking Small

One of my favorite details in the cabinet is the Alta Labs switch. It’s quiet, fanless, supports PoE, and fits the overall philosophy of the build perfectly.

The same goes for the tiny patch panel. At first, I was disappointed that it uses keystone couplers instead of punch-down blocks because I was looking forward to breaking out a fresh 110 blade - then I laughed. This little cabinet is basically a scale model of the full-size racks I spent years building and maintaining in datacenters around New York City and Santa Clara. The equipment is smaller, but the principles haven’t changed.

Label everything, keep related systems together, and plan your cable management. Design so future you can understand what present you built. Good habits don’t care how large the rack is.

Looking Back

It’s funny. I set out to buy one inexpensive mini PC and accidentally built a miniature datacenter. More importantly, I built a place where I can safely experiment.

That’s what Foxmox really is. It’s not a production environment or a showcase, it’s a workshop. Something good going on when it’s powered up. An idea being thought through, a solution being baked up to solve a problem, or maybe just a bit of Tetrinet – and then the lights go out until next time.

Twenty years ago I was learning Windows Server on a commuter train. Today I’m experimenting with Proxmox clusters, Ceph, Docker, backup strategies, automation, and infrastructure design in a cabinet that fits on an IKEA countertop under my TV. Nice.