I work out of my home office full-time. I spend a lot of time here, and so I’m used to the way things look – and sound. I was on a video call this week when something felt off. I took off my headphones and heard it.

clunk.

clunk.

One of the drives in my homelab was beginning to fail.

My Proxmox server hosts an Active Directory domain, Windows test environment, LXC containers and Docker containers. It hosts media services, ad blocking and backs up data from my family’s computers.

This “homelab” isn’t one of those half-racks full of industrial-grade servers in closets you see on YouTube. I assembled mine over the years from end-of-life, unwanted and discounted hardware. My primary server is a laptop purchased on eBay for parts, with screen burn in and missing keys. It did, however, come with 20 GB of RAM. My firewall and NAS came from thrift shops. I’d thought about upgrading it, but it serves my needs well and cost less than a used Dell desktop.

Looking at the NAS logs, I saw one drive was logging an I/O error every 30 seconds. One drive might be failing. I deactivated the drive (turns out it was one of the newer white-label drives) and replaced it with a spare I had laying around. Once let the consistency check finished, all was good.

I deactivated the failing drive and replaced it with a spare drive I had laying around. I would have set up a hot-spare, but I needed all of the bays in my NAS.

clunk.

While the NAS drive was beginning to fail, the clunk was coming from an external USB drive used to back up the NAS. The drive was sitting vertically as was designed. I turned it around so the drive lay horizontally, and the noise went away. When I was starting out in IT, we had a superstition about running spinning drives sideways, thinking it could make a head crash easier. Turns out that superstition still lives in the back of my head.

I spent the rest of the afternoon pruning backups, putting a replacement external drive on my Amazon wishlist, and re-routing cables, like you do when you run a homelab.