On Migrating a Database
Our main product at HitGrab is MouseHunt, a long-standing game that started on Facebook and eventually moved to an independent website and now a mobile app for Android and iOS.
The database, where we store everything a player sees and does, was a product of an older era and in dire need of upgrading. However, our environment was a bit complicated, mixing old elements of dedicated physical servers and newer cloud server technology.
Dealing with such an old codebase that's passed through many hands, programmers and devops alike, across the 10 years it's been around is a delicate process, to say the least. Considering the size and scope of the data (over 200 GB), and reliance we have on keeping 100% of it intact, it was a nerve-racking task to be sure. This is not only our flagship product but, realistically, our main source of revenue!
On top of this, our server host provider was performing a physical migration of the actual database server to a brand new location. This meant unavoidable downtime because the actual machine would be unplugged, loaded onto a truck, and driven to its new home. With this looming downtime (overnight, and 8 hours long!), and the aging hardware, we decided to begin some basic R&D to consider migrating to a cloud server which would allow us to upgrade the database while avoiding the major downtime all at once.
After some testing and performing some mock "dry runs", we determined it could be done! It would only need 2 hours of downtime, versus 8 hours (which would have been overnight, let me remind you), and we'd come out the other end with a better, faster, stronger database!
After a grueling week of preparation, countless dry runs, practically memorizing every single step of the process, the migration was performed and went off without a hitch! Well, for the most part. There was a minor hiccup related to how the new technology applied configuration (the rules the database follows) to the new server, but after some quick debugging (and a little sweating), it was sorted out.
If I had to do this again, I would change some of the things I tested while performing the dry runs. It was a unique situation with a lot of unknown variables (how it would behave in our strange environment, if it would perform properly when the floodgates opened and players were let back in, etc.), but a very awesome learning experience.
This was one of the scariest things I've ever done while working with servers over the last 10 years. It's exciting to be in a position where you can do something to improve the performance of a product, but it's also terrifying knowing what is at stake! However, the benefits very much outweighed the calculated risks in this case, and it shows.
MouseHunt now runs smoother and faster than it ever has, and our players have noticed it as well. I can't wait for the next opportunity to do something just as impactful and exciting!
