What are the common challenges and risks of restoring systems and data from backups?

I have encountered situations where backups were either unavailable or compromised, leading to significant recovery challenges. For example, during an incident with a financial services client, we discovered that several backups were corrupted due to a previously undetected malware infection. This highlighted the importance of regularly testing and verifying backups. We implemented a routine testing schedule, utilizing encryption and checksums to protect against tampering. Additionally, we stored backups in multiple secure locations, both on-site and off-site, to safeguard against localized failures. This approach ensured that we had reliable, recent backups available when needed, significantly improving our overall resilience.

Depending on which data or configuration you are storing in the backup, there may be security and data privacy considerations to be made for the backed up data. After all, a backup is a copy of what you had in production. To maintain this level of security, handle your backup data with the same level of security controls and access management as your production data.

Tive uma experiência em 2019, onde praticamente começou minha jornada em cyber segurança, estava fazendo o backup, diário, bem "certinho", conforme, até então, mandava o protocolo, porém, estava salvando na unidade de disco D, do servidor local. Um belo dia, chego às 8h no trabalho, lembro como se fosse hoje, do pessoal reclamando que o sistema estava fora, para minha surpresa, um ransomware estava lá, com aquele txt pedindo resgate. Moral da história, nunca salve nada no mesmo disco, sempre salve externamente e "TESTE O BACKUP". As vezes só aprendemos com os próprios erros.

Ao longo da carreira vejo dois pontos que dificultam um retorno das operações: - A falta de backup - A falta de testar esporadicamente os backups. Este é um controle que a ISO 27002 já fala a anos e muitas organizações fazem pouco caso e sofrem no momento de recuperação de um ataque.

Ter backups é essencial, mas não resolve tudo. A restauração de grandes volumes de dados, como um file server com terabytes de informação, pode demorar significativamente. Embora backups sejam cruciais para a recuperação de desastres, a velocidade de restauração também é fundamental. É importante implementar estratégias como backups incrementais ou soluções baseadas em nuvem para garantir uma restauração eficaz e ágil.

Usually the time and resources for the restoration is always underestimated. With current resources operation team need to inform management team how long and how many systems they can handle, most of the time you won't have enough resources to restore everything.

At the point of restoration, remember that you will be under extreme stress and pressure. The entire business may be reliant on each and every action that you and your team take. Rehearse this till you can do it in your sleep. Rehearse it under different conditions. Know your time to recovery, time to live. Ensure that your senior team also knows that time and that it can't change. Imagine a Formula 1 team changing a tyre. Each person knows each step, they know everything with precision. The driver knows it as well as the Team Principal. You need to be like this with your restoration. Know the environment so well that there are no questions, you have your authorizers for change fully engaged and pre-empt their engagement

In one notable instance, I worked with a healthcare organization that faced challenges restoring data due to incompatible software versions and missing updates. We addressed these issues by establishing a standard backup procedure, documenting the specific configurations, software versions, and patches used. This documentation was critical in ensuring that restored systems were consistent with the live environment. Additionally, we utilized compatible backup formats and synchronization tools to align restored data with current systems. This meticulous approach minimized disruptions and ensured that the restored systems were operationally consistent with the existing infrastructure.

Backup Integrity: Restored data may be corrupted, rendering it unusable. For e.g, if a backup file is damaged, recovery efforts may fail. Compatibility Issues: Older backups might not work with current systems. like an outdated software version could cause conflicts during restoration. Restoration can be resource-intensive, requiring significant time and bandwidth. A large database might take days to fully restore, impacting operations. Security Risks: Restored data can be vulnerable to breaches. If backups are not secured, unauthorized access could lead to data leaks. Learning from Failures: Post-incident reviews can reveal weaknesses in backup strategies. e.g, discovering gaps in backup frequency can lead to improved practices.

Tive uma experiência, pra falar a verdade, mais ou menos 3 meses, o responsável de TI atual, da empresa onde havia prestado serviços a três anos atrás, não conseguiu voltar o backup que ele havia feito, foi onde ele me ligou e perguntou se eu tinha alguma coisa da época que trabalhava lá. Eu tinha sim, salvo em nuvem e compartilhei com ele. Nessa empresa, tratava-se de uma fábrica, e tinha muita pressão para fazer tudo rápido. Lição de casa, mesmo com pressão, teste o backup.

I have found that conducting regular training sessions for IT staff on backup and restoration procedures can significantly enhance preparedness and response times. In a real-world scenario, we organized hands-on workshops where team members practiced restoring systems from backups under simulated conditions. This not only improved their technical skills but also boosted their confidence in handling real incidents. Additionally, fostering a culture of continuous improvement by encouraging feedback and suggestions from team members can lead to innovative solutions and enhancements in the backup process. This holistic approach ensures that both technical and human factors are optimized for effective backup and restoration.