thecoffeehobbit

Hi Lemmy! First post, apologies if it's not coherent :)

I have a physical home server for hosting some essential personal cloud services like smart home, phone backups, file sharing, kanban, and so. I'm looking to re-install the platform as there are some shortcomings in the first build. I loosely followed the FUTO wiki so you may recognise some of the patterns from there.

For running this thing I have a mini-pc with 3 disks, 240GB and 2x 960GB SSDs. This is at capacity, though the chassis and motherboard would in theory fit a fourth disk with some creativity, which I’m interested to make happen at some point. I also have a Raspberry Pi in the house and a separate OPNsense box for firewall/dns blocking/VPN etc that works fine as-is.

In the current setup, I have Ubuntu Server on the 240GB disk with ext4, which hosts the services in a few VMs with QEMU and does daily snapshots of the qcow2 images onto the 960GB SSDs which are set up as a mirrored zfs pool with frequent automatic snapshots. I copy the zpool contents periodically to an external disk for offsite backup. There’s also a simple samba share set up on the pool which I thought to use for syncthing and file sharing somehow. This is basically where I’m stopping to think now if what I’m doing makes sense.

Problems I have with this:

• When the 240GB disk eventually breaks (and I got it second hand so it might be whatever), I might lose up to one day of data within the services such as vikunja, since their data is located on the VMs, which are qcow2 files on the server’s boot drive and only backed up daily during the night because it requires VM shutdown. This is not okay, I want RPO of max 1 hour for the data.
• The data is currently not encrypted at rest. The threat model here is data privacy in case of theft.

Some additional design pointers:

• Should be able to reboot remotely in good weather.
• I want to avoid any unreliable or “stupid” configurations and not have insane wear on my SSDs.
• But I do want the shiny snapshotting and data integrity features of modern filesystems for especially my phone’s photo feed.
• I wish to avoid btrfs as I have already committed to zfs elsewhere in the ecosystem.
• I may want to extend the storage capacity later with mirrored HDD bulk storage.
• I don’t want to use QEMU snapshots for reaching the RPO as it seems to require guest shutdown/hibernation to be reliable and just generally isn’t made for that. I’m really trying to make use of zfs snapshots like I already do on my desktop.

My current thoughts revolve around the following - comments most welcome.

• Ditch the 240GB SSD from the system to make space for a pair of HDDs later. So, the 960GB pair would have both boot and data, somehow. (I'm open to having a separate NAS later if this is just not a good idea)
• ZFS mirror w/ zfs-auto-snapshot + ZVOLs + ext4 guests? Does this hurt the SSDs?
• Or: ext4 mdadm raid1 + qcow2 guests running zfs w/ zfs-auto-snapshot? Does this make any sense at all?
• ZFS mirror + qcow2 + ext4 guests? This destroys the SSDs, no?
• In any case, native encryption or LUKS?
• Possibly no FDE, but dataset level encryption instead if that makes it easier?
• I plan to set up unattended reboots with the Pi as key server running something like Mandos. Passphrase would be required to boot the server only if the Pi goes down as well. So, any solution must support using a key server to boot.
• What FS should the external backup drives have? I'm currently leaning into ZFS single disk pools. Ideally they should be readable with a mac or windows machine.
• Does Proxmox make things any easier compared to Ubuntu? How?
• I do need at least one VM for home assistant in any case. The rest could pretty much all run in containers though. Should I look into this more or keep the VM layer?

I'm not afraid to do some initially complex setting up. I'm a full stack web developer, not a professional sysadmin though, so advice is welcome. I don’t want to buy tons of new shit, but I’m not severely budget limited either. I’m the only admin for this system but not the only user (family setting).

What’s the 2025 way of doing this? I’m most of all looking any inspiration as to the “why”, I can figure out ways to get it done if I see the benefits.

tldr: how to best have reliable super-frequent snapshots of a home server’s data with encryption, preferably making use of zfs.