Here's my workflow for any data I can't afford to lose.

3-2-1-1 strategy

• Three (3) copies of the data (well four counting production)
• Two (2) different formats
• One (1) copy off-site (encrypted in the cloud)
• One (1) copy offline (encrypted on a secure device)

Nightly backups

Before migrating to Proxmox, incremental backups were generated using rsync in a cron job that looked like:

30 4 * * * rysnc_script.sh || alert "Backup: ERROR"

rsync_script.sh handled the configuration, and any services that needed to be stopped and restarted. In the event of a failure, alert would send a message to my notification service using curl.

Now, this is managed with Proxmox Backup Server running in a separate VM. Daily snapshots are retained for the last 30 days, while monthly snapshots are kept for the last six months.

Off-site / Offline export

Every two weeks I run a script that:

1. Creates a compressed archive (tar.gz) of the latest snapshot
2. Encrypts it using AES-256
3. Uploads the encrypted archive to the cloud
4. Notifies me the archive is ready to be moved offline

In the past this ran every week. However, I noticed most of my really important data wasn't changing much week over week. Moving to a bi-weekly schedule has been a good blend of convenience and data integrity so far.

Testing

Making sure to test my backups has been crucial. I do it in two ways:

1. Checksum validation: this is automated and runs every night, notifying me of any problems.
2. Restore drill: once a month, I spin up a new VM using my latest off-site archive and confirm everything starts correctly.

I keep a simple "restore checklist" in a repository (along with a printed copy) so anyone can follow it step-by-step.

Being able to really trust my backups has helped turn any scary situation into a routine confidence boost.

Our paper list was always getting left behind, lost, or thrown away. So I set out to digitize it. But, I didn't want to rely on some external service like dropbox or google docs to do it. Instead, the first spare computer I had sitting around got a fresh copy of Debian and away I went.

What began as a simple idea eventually grew into multiple servers, dozens of services, and endless projects.

Design

When I started to build my lab, the plan was to create something dependable. I didn't (and still don't) want to spend all my free time making sure everything simply works.

The backup plan was to have reliable backups. I drafted a policy and set off to make sure any important data would always be somewhere in case of an emergency. Six-ish years later and I still haven't managed to lose anything important. yet. *knocks on wood*

Security was and still is a major concern. I don't like people touching my stuff, so I want to make sure only those authorized can get anything in, or out. I'm constantly learning new concepts and finding ways to integrate them and improve the overall design.

Infrastructure

The backbone of my lab is the forge, which is currently powered by Forgejo. After testing and using a few other services, I migrated to Forgejo a few years ago because I really like the community and governance structure - plus it powers Codeberg where my public code is.

Within my forge, any production infrastructure is defined as code. This includes configuration files, build steps, and playbooks to deploy hosts.

Networking

Part of building something dependable was making it available and easily accessible. What began with open ports and a static IP address, quickly got replaced by a reverse proxy to handle routing and encrypting connections using TLS.

Once everything was accessible internally, the next challenge became remote access away from home. Again, open router ports were quickly replaced by VPN access for anyone deemed worthy.

The lab has always been an environment for me to learn and explore new networking concepts. Over time this has included IPv6, mTLS, VLANs, and more.

Development

After a reliable system was in place, I could begin developing the tools and services to run on it. There are lots of amazing projects I found that could be easily self-hosted, however I love learning, and writing code, so I often end up building my own stuff.

Here are just some of the things I have built or am working on:

• asset catalogues: books, hardware, digital assets, etc.
• audio transcription tool
• digital toolbox
• pastebin
• personal finance manager
• text/audio/video chat using WebRTC
• video/image sharing platforms

I made most of these just for me (plus my friends and family), but some might end up being released into the wild one day.

Documentation

Besides backups, good documentation has been the most important part of my lab. I started with handwritten notes and slowly grew them into various tutorials, how-to guides, explanations, and reference materials. More than once I've forgotten how (or why) I did something and only had notes from past me to rely on.

Although my documentation is far from perfect, it's constantly evolving to stay useful and relevant going forward.

Now

Everything is in a pretty good state. Over the last couple years I focused on reducing the size and complexity of my lab without sacrificing performance. Turns out I didn't really need much to run everything, plus I sort of enjoy the challenge of doing more with less.

Right now that consists of:

• Protectli Vault: router running OPNSense
• Intel NUC: running virtualized dev/prod Debian environments using Proxmox
• Raspberry Pi: mainly to control my sit-stand desk and a few other things

I also have a VPS running several services outside of my lab.

Next

New hardware is almost always on the radar, however I'm frugal (aka cheap) and a strong advocate of "if it ain't broke don't fix it". I'm looking for a decent UPS though, and some extra storage could always be put to good use.

With a stable system, and most of my development needs covered for now, I'm excitedly looking for the next problem to solve.

Whatever that might be.