3 is the magic number
Traditional RAID is not cutting it anymore. Large multi petabyte environments and the use of high capacity drives seem to have introduced the demise of our good old friend RAID and made us look for efficient alternatives. The obvious place to look for an alternative is at companies like Amazon, Facebook, Google, etc,.. how are they protecting their large hyper scale environments? Surely they are not using RAID technologies?
In these large web scale environments writing multiple copies of data or erasure coding seem to be popular data protection alternatives offering high availability and scalability without the limitations of traditional RAID.
Because erasure coding has a performance hit it seems to be suitable for mostly static or non latency sensitive workloads. For the majority of the workloads out there writing multiple copies of data combined with inline compression and deduplication seems to be the best alternative to RAID offering the ultimate trade off between capacity, performance and availability. So when you write multiple copies, how many copies should you write? Is 2 copies of data (original and one copy) enough, or should you go for 3 or more copies?
To find an answer to this question it makes sense to look at big data and how they cope with data protection.
The following blog article written by Ted Dunning (Chief Application Architect at MapR Technologies) gives some interesting insight into why 3 copies of data seems to be the optimal compromise between performance and reliability https://www.mapr.com/blog/revealed-why-hadoop-uses-three-replicas#.VYRG6mB_mkP
3 is the magic number - Since surviving 2 failures is the minimal reasonable protection today it would indeed make sense to write 3 copies (or more) for production data.
When writing only two copies of your data (where the first is your actual data) when one drive or storage node fails your already in a tight spot. Imagine if another drive or node in that same environment should fail at the same or almost same time? I wouldn’t want to be the architect defending why we selected this protection scheme…
Comments are welcome
