Robust Docker Backups: Mitigating Risks in Production Environments

Introduction: As a consultant, We recently had the opportunity to assist an ecommerce client who was looking for a way to backup their Docker infrastructure in production without stopping their containers.

The client emphasised the importance of minimising downtime and ensuring uninterrupted service to their customers.

In this blog post, I will share how we helped them implement a backup solution that achieved their goals while maintaining data consistency.

The Challenge:

The client's ecommerce platform relied heavily on Docker containers to run various micro-services and databases. While they understood the importance of regular backups, they were concerned about the potential impact of stopping containers on their production environment.

The client needed a solution that could backup their Docker volumes and containers without causing any service interruptions.

The Solution:

After analysing the client's infrastructure and requirements, We proposed the following backup strategy:

Filesystem-level Snapshots:

• Utilised the underlying filesystem's snapshot functionality (e.g., LVM snapshots for Linux) to create point-in-time copies of the Docker volumes.
• Implemented a script to automate the creation of snapshots on a regular basis (e.g., hourly, daily) based on the client's recovery point objective (RPO).
• Here's an example of creating an LVM snapshot using shell commands:

# Create an LVM snapshot
lvcreate --snapshot --size 10G --name snapshot_volume /dev/vg_docker/docker_volume        

Snapshot Replication:

• Configured the script to replicate the snapshots to a separate storage location, such as a backup server or cloud storage (e.g., Amazon S3).
• Utilised incremental replication techniques to optimise network bandwidth and storage space.
• Here's an example of using the AWS CLI to sync snapshots to an S3 bucket:

# Sync snapshots to S3 bucket
aws s3 sync /path/to/snapshots s3://backup-bucket/snapshots        

Retention Policy and Cost Optimisation:

• Implemented a retention policy to keep a certain number of snapshots based on the client's recovery time objective (RTO) and compliance requirements.
• Automated the deletion of older snapshots that exceeded the retention period to optimise storage utilisation and costs.
• Here's an example of using the AWS CLI to delete snapshots older than 30 days:

# Delete snapshots older than 30 days
aws s3 rm s3://backup-bucket/snapshots/ --recursive --exclude "*" --include "*-$(date -d '-30 days' '+%Y-%m-%d')*"        

Backup Verification and Testing:

• Developed a process to periodically verify the integrity of the snapshots and ensure their recoverability.
• Performed regular restore tests in a non-production environment to validate the backup and restore procedures.
• Here's an example of restoring an LVM snapshot:

# Restore an LVM snapshot
lvconvert --merge /dev/vg_docker/snapshot_volume        

Note: While the approach of using filesystem-level snapshots to backup Docker containers without stopping them can be effective, it's important to consider the following limitations and potential drawbacks:

1. Consistency Risks: Taking snapshots of a live filesystem can introduce consistency risks, especially for databases or applications that heavily rely on disk I/O. If the snapshot is taken while data is being written, it may capture an inconsistent state. It's crucial to thoroughly test the backup and restore process to ensure data integrity.
2. Performance Impact: Creating snapshots and replicating them to a remote location can impact the performance of the underlying storage system. It's important to monitor the performance during the backup process and schedule backups during off-peak hours to minimize any disruptions.
3. Large Docker Containers: This approach may not be suitable for backing up large Docker containers, as the snapshot creation and replication process can take a significant amount of time and consume substantial storage space. For large containers, it's recommended to explore alternative backup strategies, such as container-level backups or database-specific backup tools.
4. Compatibility: The snapshot functionality and commands may vary depending on the underlying filesystem and storage system. It's important to ensure compatibility and thoroughly test the backup solution in the specific environment before deploying it in production.

Future Enhancements: To further improve the backup solution and address some of the limitations mentioned above, we are planning to explore the use of "btrbk" (https://github.com/digint/btrbk) in the future. "btrbk" is a backup tool specifically designed for Btrfs and other copy-on-write filesystems, which can provide more advanced features and reliability compared to custom scripts.

By leveraging "btrbk," we aim to achieve the following benefits:

• Improved consistency and reliability of backups
• Built-in support for incremental and differential backups
• Efficient handling of large Docker containers
• Enhanced performance and resource utilisation
• Simplified configuration and management of backup tasks

We will thoroughly evaluate "btrbk" and conduct extensive testing to ensure its suitability for our client's environment. Once validated, we plan to integrate "btrbk" into our backup strategy to provide a more robust and reliable solution for backing up Docker containers in production.

Conclusion:

As we continuously strive to enhance our backup solution, exploring the use of specialised tools like "btrbk" can help us address the limitations and provide a more robust and reliable approach to backing up Docker containers in production.

In conclusion, implementing a comprehensive and reliable backup strategy for Docker containers is essential for ensuring data protection, business continuity, and peace of mind. By staying updated with the latest tools and best practices, we can evolve our backup solution to meet the ever-changing needs of our clients and their critical applications.