Automating Backup Processes: Tools and Best Practices

Backup checks. IDGAF how many automated restore checks run or that the backup logs say successful. I’ll automate the shit out of our processes, but for Backup and DR, we’ll regularly, manually review our backup configurations to ensure all servers and volumes are protected, regularly, manually verify our restore processes, and manually, routinely test our BCP. We can fuck up just about anything, and a whole lot can go wrong, so long as our backup and DR is sound, we’ll survive.

(Yes, in addition to automated alerting and automated tests and verifications).

We update data several times a day directly through SQL scripts. Management wants to automate a folder that developers can drop scripts into and a cron job will pick them up. Sure, let’s just have a process that runs random SQL code against prod 10 times a day and see how that goes.

My first cup of coffee in the morning

I’m not asking what could be automated in theory.

I’m asking what you intentionally keep manual because when it breaks, the blast radius is too big.

Every system I’ve seen has at least one process like this, usually held together by habit, fear, or undocumented edge cases.

Curious what that process is for you, and why it hasn’t been touched.

6 year streak broken. My first post.

I built (vibed) vmbackup — a Bash-based backup manager that handles the stuff virtnbdbackup doesn't: scheduling, rotation, retention, replication and reporting.

What it does:

• Discovers all your VMs via libvirt — running, shut off, paused — and backs them up with the right method for each state (FSFREEZE for running VMs with a guest agent, pause for those without, copy for shut off)
• Policy-based rotation — daily, weekly, monthly or accumulate. Each VM can have its own policy
• Self-healing — if an incremental fails it converts to full and retries. Broken backup sequences get archived and restarted automatically. Stale locks from interrupted runs are cleaned up on the next run
• Replication to NFS and cloud (contract based, write - or vibe - your own!)
• TPM and BitLocker support — backs up TPM state and can extract BitLocker recovery keys via the QEMU guest agent
• Email reports after each run
• SQLite database for backup state tracking
• No Python, no database server, no web UI. Just Bash, sqlite3 and jq. Minimal dependencies.

GitHub: 

vmrestore is next.

vmrestore:

I have a number of sites running on the same server, and I want to set up an automated task that will export every database once a week into a specific folder, and then somehow get them into my version control system.

The best I've turned up in google is suggestion of cron jobs. I could set up a cron job for every database individually, but even then I'm not sure how I would get the files committed into version control without human interaction.

Does anybody know of the best way to do this?

edit: some very interesting responses so far. thanks to everyone for contributing.

After a pretty disastrous data loss when a hard drive corrupted, I learned the real truth that raid is not the same as backup. So now I'm doing something about it. I am using my old Synology NAS as a destination to do weekly backups from my Ugreen DXP 4800 pro.

I thought rsync was the answer, but I realize that it creates .ubk file that isn't searchable. Really what I want is to have a weekly updated mirror image of my Ugreen folder structure. What is the easiest way to do this?

So I don't know much about this subreddit or about data storage in general, but I'm trying to figure out how to keep my automatic data backup system (which is through Backblaze) from just uploading corrupt or damaged files to my cloud-backup directly from the primary archive (which is just a big hard drive that sits under my desk). Is there any way to know when a file on the drive becomes corrupted so it can be replaced with the backup version of the file without having to manually search through and validate every single file in the archive (that would take an eternity given how much stuff I have stored). How can I most effectively protect against corrupted or damaged files? (I've already found more than I'd like that I didn't know were busted and I really want to protect the rest if at all possible. Any help would be greatly appreciated!

I'm pretty new to opnsense, but I don't see any option in the UI or an API endpoint to create a backup, only to download backups that have been manually created. I have a simple use case where I want to generate backups and download them to my local NAS, is that possible? Thanks in advance.

Edit: so from extrapolating from the comments that there is no need to generate a backup as the backup api endpoint just grabs the latest version of the config. For anyone in the future looking for similar information, here is what I went with:

• Create a backup user with the Diagnostics: Configuration History privilege
• Create an API key for the backup user
• Create a simple backup script that pulls the config using the API

```

api_key="[ api_key ]" api_secret="[ api_secret ]" opnsense=" opnsense_ip ]" endpoint="/api/core/backup/download/this" out_dir=/opnsense_backups

curl -k -u $apikey:$api_secret $opnsense$endpoint -o $out_dir/opnsense-backup-$(date +%F%H:%M).xml >> /var/log/cron.log 2>&1 ```

• Create a cron container that performs a backup daily (or however often you'd like)

```

FROM debian:stable-slim RUN apt-get update && apt-get install -y cron curl && rm -rf /var/lib/apt/lists/* COPY crontab /etc/crontab COPY opnsense_backup.sh / RUN touch /var/log/cron.log CMD ["cron", "-f", "-L", "15"] ```

```

SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin

00 00 * * * root /opnsense_backup.sh ```

podman build -t cron .

```

podman run -d -v /data:/opnsense_backups \ --restart=unless-stopped --name cron localhost/cron ```

Hello, so I have a VPS and I lost the data before due to user error, and I never want this to happen again. So, is there a way I can make automated backups to Google Drive, Dropbox? Or maybe even OneDrive? I have the auto-backup add-on through my hosting provider, Contabo, for daily backups on that end, but I would like another fail-safe if that doesn't work for whatever reason.

Hey folks,

I’ve recently joined a team where deployments are still fully manual, runbook-driven, and pretty error-prone. I’ve been asked to look into automating the process

I should also mention I’m fairly new to this, so I’m trying to be thoughtful about not overengineering things or picking the wrong approach early.

We have two apps:

Market-facing app on Kubernetes (EKS on AWS)

Integration app on ECS (Docker-based)

Two environments: demo and production. I’m planning to automate demo first and only touch prod once things are proven.

Each deployment is a long sequence of manual steps, roughly:

Pre-checks (current version, data reconciliation)

Backup + verify it’s safely in S3

Stop services

Pull and configure new release

Run upgrade

Post-checks (pods healthy, UI version correct)

Notify team + scale down

The integration app differs a bit:

Pull from Git

Build Docker images

Force deploy to ECS

Also worth noting:

Some deployments are full upgrades, others are patches, and the steps differ meaningfully

I want to turn this into a reliable pipeline instead of relying on someone executing 30+ steps perfectly every time.

A few things I’m unsure about:

1. Tooling

We’re already deep in AWS. For a mixed EKS + ECS setup, would you lean toward:

CodePipeline / CodeBuild

GitHub Actions

Jenkins

Something else

2. Pipeline design

Would you:

Build one parameterized pipeline

Or split by app and/or environment

Right now I’m leaning toward separate pipelines per app, but curious what’s worked (or failed) for others.

3. Approval / safety gates

Some steps need human confirmation, especially backups.

Example: we should not proceed unless someone confirms the backup completed successfully.

What’s the cleanest way you’ve implemented this?

Manual approval steps in pipeline tools

External checks

Something else

4. Notifications

We currently send MS Teams messages at start/end of deployments.

Would you:

Integrate notifications into the pipeline

Or keep that separate

If you’ve built something similar, I’d really appreciate any advice, patterns, or horror stories. Especially around what not to do.

Thanks! 👊🏻

Is there a way to automate backups to another cloud storage provider instead of dropbox?

In an ideal world I would like to automate my back up to my pCloud account.

I use a fully automated DIY backup setup for my VPS that backs up both my Astro-based websites and a PostgreSQL database from self-hosted Umami analytics. It’s been far more reliable than relying solely on my hosting provider’s daily backups.

Here's the architecture diagram:

My approach uses a Bash script scheduled via a systemd timer. It runs daily, creates compressed backups of the site directories (excluding stuff like node_modules, dist, etc.), and dumps the Umami database using pg_dump from within a Podman container.

The real magic comes from rclone. It syncs the backups to three different cloud services: Box (primary), pCloud, and Koofr. Old backups over 60 days get purged automatically to save space. Everything is fully unattended once set up—true "set it and forget it".

Since my containers bind their app folders to host directories, the script can access the source files directly without interacting with the containers. The result: small, fast, and redundant backups.

Setting up rclone for each cloud provider takes a bit of time, but once it’s running, you never have to think about it again. Definitely more peace of mind than just relying on your VPS provider’s backup solution.

I have an automated daily backup with Kopia to Backblaze. It makes incremental snapshots. I sometimes check if they work if I feel like it. Kopia lets you mount a snapshot to a directory and browse, it only loads the stuff you actually access so it doesn’t take your download “credits”. I don’t do local as I don’t need my data right away if the server goes down. And it’s pretty unlikely both my server and Backblaze are down at the same time (it’s a gamble lol). I backup critical stuff, no point backing up caches or generated data. With Jellyfin it’s gigabytes of thumbnails. I have around 300GB and it’s around $1.5 a month or something

Windows or Linux? What control panel if any ?

Kopia all the way in "local repo" and sync once every night to blackblaze server.
(in some word to describe kopia IMO : Entreprise backup solution, for selfhosting.)
That shit is craaaazy

I just self-host a web server so my backup requirements for that are fairly small. As I use Windows, I just use robocopy and copy the files to a couple of external drives. I make two copies of the files; one is for a clean reinstall and the other is everything that I've ever created.

I could run the backups in Task Scheduler, but I prefer not to keep the external drives attached to my computers, so I run the backups manually. That's just done weekly or daily, or whenever I feel like doing it.

Making sure the backups were done properly is just a matter of looking through the robocopy logs as it is very good as self-reporting.

I backup everything I have had a hand in creating, files, logs, scripts and so on. Any program files can be easily downloaded again but I also backup the downloads folder in case they are not.

I keep two copies of the files on external drives as well as whatever is on OneDrive and Google Drive

How I do my backups may not be for everyone, but it works for me. I recently changed the computer I use as a web server and the backups worked exactly as they should.

Did you check with your VPS provider? They should offer an easy automatic solution that doesn’t require any setup on your side.

im using duplicacy in docker. but there are various other tools out there.

And yes: I really think it's time to get a generic Backup solution from UGreen.

I was running into the same issue and had to create an rsync script to get it to work. I've left instructions as a response to my own inquiry here, maybe it will also help you.

Good luck!

I backup to external USB HDD. Incremental and daily.

I'm self-hosting several applications and I realize my backup strategy is basically "I hope nothing breaks." I need something more reliable but not overly complicated.

Current situation:

I'm doing periodic manual backups to an external drive. It works until it doesn't. I'm worried about:

• Forgetting to backup
• Not noticing if backups fail
• Losing data if the server and backup drive both fail
• Restoring from backups when I need them

Questions I have:

• Do you automate backups, or still do manual backups?
• What backup frequency makes sense (daily, weekly)?
• Do you test restores, or just trust they work?
• What's your backup storage strategy (local, cloud, both)?
• How do you handle incremental vs full backups?
• What do you back up (everything or just critical data)?
• How much are you spending on backup storage?

What I'm trying to achieve:

• Automated backups I don't have to think about
• Confidence that I could recover from failures
• Not spending a fortune on storage
• Simple enough to maintain long-term

What's a realistic, sustainable approach?id