The Evolution Of Data Storage

Introduction:

            Every organization is investing billions of dollars to build a robust solution to meet the performance requirement of their businesses for a seamless customer experience. The storage industry has seen a lot of development around performance, latency, IOPS. The expectations from data storage devices are going up every single day. Giant storage vendors are investing a huge amount in developing storage technology. However, it is important to understand how the storage development has taken place over the years. What made the industry develop new types of drives and storage communication protocols. Was it even required?

HDD (Hard Disk Drive)

            Remember the early days of storage technology when the data was stored on HDDs. HDDs were introduced by IBM in the 1950s. In the beginning, the capacity of a single HDD was around 3-5 MBs. The performance of a hard drive is most effectively measured by how fast data can be transferred from the spinning media (platters) through the read/write head and passed to a host computer. The platters need to spin faster to increase performance on a hard drive. 

It was conducive that more moving parts higher will be the failure rate of the drives and therefore HDDs tend to fail a lot. Apparently, there was a need for a storage drive that can endure more. That’s when the industry came up with a new type of drive i.e., SSD.

SSD (Solid State Drive)

SSDs got their name ‘solid state’ because they have no moving parts. In an SSD, all data is stored in integrated circuits. SSDs replace traditional mechanical hard disks by using flash-based memory, which is significantly faster. SSDs speed up computers significantly as they have low read-access time and better throughput. They use a NAND flash memory which has near-instant access times.

Solid State Drives, enabled by NAND technology, have the potential to provide much higher performance (IOPS and MB/s) but, since the data was actually flowing on SAS protocol, the complete benefit of performance from SSD drives was not utilized because of the limitations of SAS protocol in handling no of commands at the same time and therefore to leverage comprehensive benefits from SSDs, the need of faster communication protocol was needed. To cater to this need, the industry introduces the NVMe protocol on storage arrays.

Serial Attached SCSI (SAS) vs Non-volatile Memory Express (NVMe)

When the external storage arrays were introduced, it needed a way to attach more numbers of drives to the CPUs while maintaining the use of the SCSI standard. This was the reason why Serial Attached (SCSI) SAS was developed. This development helped to attach numerous drives to a shared bus, While NVMe is a storage protocol that is designed to take advantage of modern high-performance storage media. 

NVMe offers a parallel and scalable interface designed to reduce latencies and increase IOPS and bandwidth. It supports more than 64,000 queues and 64,000 commands per queue which is drastically higher than the SAS which supports a single queue and 32 commands per queue. 

The data-driven applications are performance-hungry and they are computing-intensive workloads. With more queues, the storage devices can process and complete IO requests even faster in fewer drives than SAS. 

Therefore, it is imperative for any organization to understand what types of applications and workloads are going to be deployed on the storage and what level of performance and throughput is expected from the external storage device. It is a decisive factor for the type of drives and communication protocol.

Disclaimer

I am a Technical Specialist at IBM at the time of posting this article but any views, opinions presented here are my personal. They don’t necessarily represent my organization’s positions, strategies, or opinions.