How often do you test and verify your server backups?

In my role as a hosting provider, managing backups is paramount due to the reliance on data across multiple servers hosted both within our Data Center (DC) and in the cloud. Our approach to backup solutions is multifaceted, tailored to ensure data security and integrity at all times. We leverage Nutanix for our DC backups, utilizing its Snapshot functionality to capture server states efficiently. This method allows us to quickly revert to a previous state in case of an issue, minimizing downtime and data loss. We have Veeam Backup & Replication for comprehensive protection. It performs incremental backups daily, weekly full backups, This dual-layered approach not only safeguards against data loss but also optimizes storage efficiency.

The frequency of backups should align with your data's value and how often it changes. Daily backups are standard for critical systems, but less dynamic content might need only weekly or monthly saves. The trick is balancing risk and resource investment. You have 3 options. Full backups capture everything, offering an easy restore process but requiring more storage and time. Incremental backups are more efficient, saving only data changed since the last backup, reducing time and space but complicating restores. Differential backups strike a middle ground, capturing changes since the last full backup, offering a balance between efficiency and restore complexity. Choosing the right mix and frequency is key to a resilient backup strategy.

Critical files should be backed up daily, if not hourly. A critical production database, doing a daily full backup with hourly differential backups will ensure that at least if the system goes down you will only use the last hour of information. Less important files can be backed up on a weekly basis. I don't see why you'd want to only perform a backup once per month. I generally use a 3-4 week retention period for backups made with the same filename to manage cloud storage fees. At least that way, if an issue is identified during that timeframe we can restore the data.

The frequency of backups depends on how often the data mutates and how much you can afford to lose in between backups. For example for a simple WordPress site, daily backups of the docroot will likely suffice (thanks to ZFS and snapshots we can do this 4 times a day) For more critical data like production PostgreSQL databases you run a weekly / daily full backup and archive the WAL's, allowing you to do a point in time recovery as required up to the minute (or up to the transaction in case of synchronous replication)

The 3-2-1-1 backup strategy is an extension of the traditional 3-2-1 backup rule. It provides an additional layer of protection against data loss by incorporating immutable storage. Here's a breakdown of each component and its role in the strategy: 3-2-1-1 Backup Strategy 3 Copies of Your Data: Keep at least three copies of your data: the original data plus two backups. 2 Different Storage Types: Store the backups on at least two different types of storage media (HD, SSDs, tapes,etc) 1 Off-Site Copy: Ensure one of these backups is kept off-site. 1 Immutable Copy: Keep at least one backup copy in an immutable storage format. Immutable backups cannot be altered or deleted, protecting against ransomware attacks and accidental modifications.

Over my 20-year journey in IT, I've learned that where and how you store backups are as crucial as the backups themselves. Diversifying backup media and locations safeguards against data loss from hardware failure, natural disasters, or cyberattacks. Traditionally, we've used tapes, hard drives, and optical media. Nowadays, cloud storage is indispensable for its scalability and off-site security. Each medium has its merits. Tapes are cost-effective for long-term storage, hard drives offer speed for quick restores, and the cloud provides accessibility and disaster recovery benefits. Keeping up to 3-2-1 backup rule ensures that even if one backup fails or is compromised, you can recover your data and maintain business continuity.

Following the 3-2-1 backup strategy makes sense; What is often overlooked is how fast you can recover. In that sense it makes sense to have one backup option "close" to the data to ensure rapid recovery. Also important is to make sure one of the the backup options is immutable; This can be your off-site backup, stored as tape or more modernly.. object storage like S3 glacier

For Linux, rclone is a good tool for backing up the data to offsite storage and supports a range of different storage provider systems. The drawback is that it's a cloning tool; however, just pushing the files to the cloud storage where a retention policy removes old versions after a period of time works good enough for me.

For simple file based backups, rsync to a ZFS storage machine with snapshots allows easy reverts as required. For PostgreSQL databases some tool like Barman allows you to create full backups and recover up to the last replicated and applied statement. For more advanced usecases we are evaluating Proxmox backup server as it's open source with support licenses and integrates well with Proxmox VE to give you compressed, deduplicated encrypted backups

I always got to remember our product's RTO (Recovery Time Objective) and RPO (Recovery Point Objective) which is crucial in disaster recovery planning. RTO is the target time to recover operations after a disruption, indicating how quickly you aim to have systems back up. If your RTO is 2 hours, you plan to restore operations within 2 hours after an outage. RPO defines the acceptable amount of data loss measured in time. For example, an RPO of 30 minutes means you can tolerate losing up to 30 minutes of data. These parameters help set the standards for backup frequency and recovery speed, balancing the need for business continuity against the costs and capabilities of your disaster recovery solutions.

Often missed, but very important. Every backup tool you use should at the very least report succes or failure (could be as simple as an exit code). I generally appreciate the tools to only mail on issues (the Unix rule, silence is golden applies) and even simple shellscripts running in cron can be monitored using webhooks.

You should run a test backup AND recovery when setting the project up and possibly retest this at some intervals. This can be as simple as running a full database recovery to a new VM to test for a disaster. Or recovering a bit of the data, like a table that had an update without a where clause :-) For example, once per 6 months is often used. With more modern approaches like Kubernetes and Operators it becomes relatively easy to recover a backup for testing; Automating the test itself becomes the hard part.