Hi OP, I am in a similar situation as you were so I wonder which solution you chose at the end and if you are happy with it?

Buy external drives. Don’t run them in RAID, use one to store backups and plug it in once or twice a week to copy data to it.

The secret to RAID is that it doesn’t buy you data protection, it buys you uptime to access data while a device in the array is failed. This is most valuable to businesses that can’t afford the downtime that recovery from a backup incurs. The most paranoid RAID will still fail sooner or later, due to hardware or software failure, and as a home user with a limited budget you’re far better off having one offline backup that you can use to recover data from once that happens.

Backup only data you can’t afford to lose (eg: don’t backup downloaded data that can be replaced easily, like a game or movie collection) and your backups will be much more manageably sized and you won’t need to spend as much on your backup drive. If a backup disk is too much for your budget you can always exploit cloud backup plans, backblaze PC backup has no limit on the size of your backups and only charges something like ~$60/yr.

Edit: It’s also worth thinking about what kind of data you’re storing and splitting that data across multiple devices if possible. If you’re storing bulk data where performance isn’t critical, like backups from other machines or a movie collection, you can pay a much lower price by buying a hard drive instead of flash. Even if only some of your data requires fast flash you can still use a cheaper HDD to store bulk data and buy a smaller flash drive for performance sensitive tasks. When I build NAS I split my data two pools, one bulk pool of HDDs and one much smaller fast pool comprised of flash storage. Put performance critical data on flash, put bulk storage on HDDs, this will allow you to spend less on bulk and still have fast storage performance for tasks that require it. A 512GB or 1TB SSD alongside a 4TB, 6TB or 8TB HDD is significantly cheaper than spending on a 4TB or 8TB SSD.

Shop eBay for refurbished storage, it’ll be significantly cheaper than spending on brand new drives.

https://www.linuxserver.io/blog/2017-06-24-the-perfect-media-server-2017

I did perfect media server It’s got mergerfs for splitting data and using disks in various sizes .and snapraid for a level of redundancy. Tho raid isn’t backup.

That said I’m now running this setup on a n100 machine with a qnap tl-800c jbod USB c box.

Works great for downloads / Plex and home server needs.

The b100 chip isn’t amazing… Don’t get me wrong but it works really well for Plex.

Hope this all makes sense. I’m on mobile with out my glasses. Lol

Drives connected to usb have an unstable connection in my experience, this is very annoying and gets worse with hubs.

RAIDs reduce the time a system is offline and reduce data loss, if a drive fails and you can afford to wait for the new disk and the backup to restore, and have regular backups that ensure no important data gets lost (though remember the data added between backups may be lost) then you don’t need a RAID.

I don’t use RAIDs cause if my disk fails then I can stomach the 2-4 days it takes to buy a new one and restore the backup

Very important: use S.M.A.R.T and a filesystem with checksums to make sure you’re not backing up corrupted data and know to get a new one

For encryption at rest you may want to look at clevis and tang, though you need a server in your home network for this to work. The client (with clevis) then decrypts the disk at boot if it can reach the server (tang). The server can’t decrypt the data without the client secret and the client can’t decrypt it without the server public key.

Don’t know what your server could be though, maybe a router with custom firmware?

You should also look into cloud storage/rclone, that way you can automate your backups more and reduce the need for manual intervention.

I use rclone and restic to automatically backup my servers daily which takes a few seconds most of the time due to them being incremental backups.

Love the effort you’ve put into this question. You’ve clearly done some quality research and thinking.

When I asked myself this same question a couple of years ago, I ended up just buying a second hand Synology NAS to use alongside my mini-pc. That would meet your criteria, and avoids the (I’m not sure what magnitude) reliability risk of using disks connected over USB. It’s more proprietary than I’d like, but it’s battle tested and reliable for me.

I personally had the best experience with mergerfs (Drives can be any size and can be backuped by snapraid) and an external enclosure up until recently. Unfortunately USB is such a limiting factor because of bandwidth and also latency. I can only realy recommend to get a new cheap Server which has Support for sata If you want any usability while Transferring or moving files.

For automatically unlock encrypted drives I followed the approach described in https://michael.stapelberg.ch/posts/2023-10-25-my-all-flash-zfs-network-storage-build/#auto-crypto-unlock

The password is split half in the server itself and half in a file on the web. During boot the server retrieves the second half via http, concatenates the two halves and use the result to unlock the drive. In this way I can always remove the online key and block the automatic decryption.

Another approach that I’ve considered was to store the decryption keys on a USB drive connected with a long extension cable. The idea is that if someone will steal your server likely won’t bother to get the cables too.

TPM is a different beast I didn’t study yet, but my understand is that it protects you in case someone steals your drives or tries to read them from another computer. But as long as they are on your server it will always decrypt them automatically. Therefore you delegate the safety of your data to all the software that starts on boot: your photos may still be fully encrypted at rest so a thief cannot get them out from the disk directly, but if you have an open smb share they can just boot your stolen server and get them out from there

Oh man, what a mess. It is just not worth it if you’re only adding 1 or 2 TB. Also you don’t say what kind of data you want to store on this system. If it’s media files (static once written) that can simplify things.

I’d say don’t mess with external drives at all. Your simplest path is upgrade your 1TB internal SSD to 2TB or 4TB. Those aren’t too expensive, and you get SSD storage. Yes you may as well use LUKS unless you want to get fancier. I have some thoughts about key management but haven’t implemented them in practice, so talk about that would be theoretical.

RAID is for when you have data that changes, like databases where you frequently add rows or do updates, so you are up to date if a drive crashes just after an update. It also lets you keep the system running while you hot swap the crashed drive. If you don’t mind taking your storage offline while you restore from a backup, and you don’t mind having to recreate the most recent data, you don’t need RAID.

I simply keep my static stuff and backups on a Hetzner StorageBox, encrypted with Borg Backup. That eliminates all the hassles of RAID, buying hardware and keeping it at home, etc. I can remote mount it (read only) with sshfs with all cryptography happening on the client side (in practice I don’t do that very often). There’s no need to use an encrypted file system on the server, or for the server to ever see plaintext. Of course StorageBox is not self hosted, but you could do something similar with a bare iron storage server. Anyway I think it’s difficult to beat this for economy until you have tens or maybe 100’s of TB of data.

You’ve clearly done your homework, and you’ve gotten a lot of good feedback already, so I’ll just add a few points…

• Storage options: Personally, I’d replace the existing drive with the highest capacity I could afford. In an ideal situation, I’d keep the host on another drive (NVMe or flash) and dedicate the large drive to a single partition of data storage.

  In my own mini-PC (8th gen NUC), I’ve got a smaller NVMe for Proxmox and a single 8TB internal SSD for data.

• Encryption: If you’re going to bother with encryption, I wouldn’t half-ass it. Why bother at all if you’re fine using auto-decryption or a weak password that will be guessed with any sizeable effort? Just lock it down with a strong password and decrypt/mount the data drive after any reboot; making a shell alias or script for this is trivial. You’re likely not rebooting the server more than once a week anyway.

• Budget/Specs: I get the sense you don’t have much budget right now, but knowing your hardware would help in suggesting solutions. Do you have an NVMe slot? What is the make/model of the motherboard and case?

• Filesystem: For simple storage, this really doesn’t matter and Ext4 will probably be fine. It’s a mature, robust, no-frills filesystem which is perfect for bulk file storage (docs, music, videos, etc.), but Btrfs would be fine too if you want more options.

• USB Docking Stations: I’ve had really good experiences with USB docking stations like this one, and I currently use it for attaching my backup HDDs each month. I wouldn’t want to rely on them for realtime data access, but they do work wonderfully for backups and one-off drive access.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

6 acronyms in this thread; the most compressed thread commented on today has 8 acronyms.

[Thread #920 for this sub, first seen 14th Aug 2024, 10:45] [FAQ] [Full list] [Contact] [Source code]