Streamlining Data Migration from Aurora Serverless V1 to V2 with Terraform and Jenkins-based approach

AWS's announcement to retire Aurora Serverless V1 in February 2023 marked a pivotal moment for cloud database management. With this change, transitioning to Aurora Serverless V2 became imperative for maintaining efficiency and ensuring minimal downtime during upgrades. This article walks you through a robust, Terraform and Jenkins-based approach to seamlessly migrate from Aurora Serverless V1 to V2.

The Challenge and Solution

The primary challenge in upgrading from Aurora Serverless V1 to V2 is managing the transition with minimal disruption to your production environment. AWS recommends a Blue/Green deployment strategy, where a new environment (green) is created to implement changes, while the current environment (blue) remains active. However, this strategy generates a new DB cluster using AWS CLI (Prerequisite) which is independent of Terraform and Jenkins, complicating real-time resource management.

To address this, our solution involves creating a new Aurora Serverless V2 DB cluster using Terraform and migrating data via snapshots from the V1 cluster. This approach, while requiring a longer deployment window, offers an easy rollback to V1 if necessary.

Migration Steps

1. Snapshot Creation: Manually create a snapshot of your existing Aurora Serverless V1 database using the AWS CLI.
2. Database Creation: Utilize Terraform to create a new Aurora Serverless V2 database from the snapshot.
3. Retire V1 Database: After successful deployment and testing, retire the old V1 database and any associated snapshots.

Key Considerations

• Prerequisites: Ensure code compatibility for V1 and V2, manage access for stakeholders, and prepare for extended downtime.
• Pre-Migration: Capture and validate V1 database data, preventing any updates during migration.
• Migration Process: Create the V2 database using the V1 snapshot, ensuring no interruptions by extending IAM role tokens to 2-3 hours.
• Post-Migration: Implement a rollback plan and validate data between V1 and V2 databases to ensure a seamless transition.
• Production Downtime: Plan for potential downtime of 2-3 hours for data sets around 40 GB, based on previous experiences.

Implementation Details

Architecture and Development:

• Jenkins Pipeline: Utilize Jenkins for quick rollbacks and to manage environment variables for database configurations.
• Secrets Manager: Configure AWS Secrets Manager via Terraform to securely store database configurations.
• Common Code: Develop reusable Python functions for database interactions, applicable across various AWS services like Lambda and Glue.

Migration Execution:

• Pre-Migration: Isolate the V1 database, restrict access, and prepare data for migration.
• Snapshot and Upgrade: Use the AWS CLI to create a V1 snapshot, then upgrade and restore it to create the V2 database using Terraform.
• Data Migration: Deploy the V2 database, validate it against the V1 data, and ensure all configurations are updated for the new environment.

Post-Migration and Rollback:

• Validation: Thoroughly validate the V2 database using automation scripts, ensuring data integrity and functionality.
• Rollback: If issues arise, rollback to V1 by rebuilding the Jenkins pipeline with the V1 configuration.

Pre-Decommissioning:

• Once migration is verified and successful, read database configuration from Terraform code. 

Decommissioning:

• V1 Database: Decommission the V1 database using Terraform scripts to remove old resources and snapshots.
• Jenkins Pipeline: Remove database configurations from pipeline.

Conclusion

Migrating from Aurora Serverless V1 to V2 requires careful planning and execution to minimize downtime and ensure data integrity. By following a structured Terraform-based approach, you can effectively manage this transition, leveraging AWS and Terraform tools to streamline the process and ensure a robust, scalable database environment.

For a more detailed guide and technical references, please refer to the AWS blog on Aurora migration.

This approach ensures that your migration is efficient, secure, and well-documented, providing a seamless transition to Aurora Serverless V2 while maintaining control over your cloud infrastructure.

About the Author: Sambit Kumar Patra is a seasoned cloud solutions architect with a passion for innovative cloud migrations and database management strategies. Connect with Sambit on LinkedIn.