My Proxmox VE 9.0 upgrade went smoothly, I ran the following command from a shell:

apt update; apt upgrade;pve8to9

Then, updated my /etc/apt/sources.list to point to “Trixie” repositories, then ran:

apt dist-upgrade

to run the upgrade process. Admittedly, I’m not running Ceph or Proxmox Backup server, it’s definitely worth checking out the update documentation at https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#In-place_upgrade.

A quick reboot, just because, and everything was up and running.

A week later, unrelated to the upgrade, I noticed that my secondary PVE server was unresponsive. I saw lots of “read error on drive /dev/sda” messages, and rebooted the server. BIOS complained the boot drive was unavailable. I checked the cables, all seemed fine.Still no boot.

Unfortunately, I’d moved the BBS from the primary server to the secondary server when I did maintenance on the primary, and forgot to move it back. The two VMs running on the secondary were the BBS and Proxmox Backup Server.

I installed a fresh copy of Windows 11 and copied by daily BBS backup (a file backup to my NAS)  to it, got the BBS working.

I took another look at the secondary, reseated everything and now it booted. The PVE gui came up, but the VMs were unavailable.

smartctl -a /dev/sda

didn’t pull up anything out of the ordinary, no remapped sectors, moderate power-up time.

I saw the message:

TASK ERROR: activating LV 'pve/data' failed: Check of pool pve/data failed (status:64). Manual repair required!

Looking at all of the LVM commands, the logical volumes all looked OK. Searching on the web revealed the command:

lvconvert --repair pve/data

After I ran the lvconvert command, the VMs appeared in PVE just fine. I copied the data files from the new BBS VM to the old BBS, and all is back up and running.

This brought up an issue with Proxmox Backup server – since it needs a VM to run, if the host running PBS crashes, how do you restore it? I wasn’t sure if the VM stored metadata in the VM or on the backup media. Hopefully the latter. While the deduplication is nice (the BBS file area is 11 gigabytes and rarely changes) being able to restore a VM directly from any Proxmox VE server is nice. I’ll have to think about what to do in the future.

I suppose I could use the Proxmox built-in backup tool to backup PBS, and PBS to back up everything else.Then, restore PBS from backup (a 2-click process) and restore everything else from PBS.

I run a 2 node cluster without Ceph and HA. One possibility is to add a third node to create a proper quorum and run Proxmox Backup server on that node. If backup metadata is stored on the target media, then an occasional  drive clone would suffice.