Much has been written about the digital economy and how it is driving an ever increasing rise in the amount of data being generated. This is leading to what some are calling the Zettabyte (ZB) Era. As an industry, we do not suffer from a shortage of staggering – often shocking – projections about the future growth in the use of data, ever-increasing broadband data rates and video content distribution. Not only have we seen the meteoric rise of video streaming, social media sharing and Internet gaming, we have also witnessed how these strain communications and storage networks. Just as rapidly, we are seeing what were niche market applications – such as HD video surveillance, virtual reality, 4K video and the industrial IoT – develop into bona fide commercial markets.
In the not so distant past, traditionally structured data, documents and business information could be stored and processed relatively easily in relational data bases. But that is no longer the case with the explosion of user-generated video from social media networks and the growth wave of many new forms of unstructured data – from satellite imagery, radar, sonar and other scientific sources, and from persistent, always-on security and surveillance applications.
“We are seeing NVMe SSDs emerge as a mainstream technology for high-performance data centre storage and the industry is now on a fast track to the flash-first data centre.” Paul Pickle |
New industry innovations are being commercialised to store and process these new waves of unstructured data. Developers have been productising software defined storage (SDS) and other evolutionary networking technologies in order to scale bandwidth or to improve quality of service (QoS). In the storage realm, these trends are now driving system architectures towards a flash-first approach.
Flash storage is everywhere
The industry is currently seeing all types of storage systems adopting flash – from the largest hyperscale deployments to traditional enterprise storage systems. To accelerate application performance at the lowest total cost of ownership (TCO), lowest latency and lowest power, flash technology is now found across nearly all storage infrastructures. We clearly see this trend represented as solid state drives (SSDs) overtake high-performance hard disk drives (HDDs). Not only are SSDs used for big data analytic workloads, but also in virtualised environments. Most recently, the industry has been defining a new approach for connecting and communicating with flash media in a new specification called non-volatile memory (NVM) express, or NVMe.
The work of the NVM Express standards group has been the catalyst, driving a universal acceptance of non-volatile memory, leveraging the universal PCIe host controller interface and optimising low latency performance by removing legacy layers of the traditional storage software stack.
We are seeing these NVMe SSDs emerge as a mainstream technology for high-performance data centre storage and the industry is now on a fast track to the flash-first data centre. Here’s why:
- Open source has hastened the development of flash-first storage software
Most public or private cloud data centres built at scale today use open source software. The various permutations of SDS solutions in the market were developed specifically to store and process unstructured data and were the earliest users of flash technology.
Most technical initiatives within these open source communities are targeted at enhancing NVMe based flash deployments. The open source communities are working across the stack, enabling new deployment options such as Ethernet-attached storage (NVMe-oF) or new application interfaces that displace traditional file systems and block interfaces.
These developers are continuing to drive the software layers within the flash device firmware to be more programmable with the objective of dynamically enhancing the performance of the diverse workloads across the data centre. Enterprise and cloud-based applications are even being designed to leverage the efficiencies of flash-based storage. Software is driving the pace of innovation for the flash ecosystem and is accelerating its adoption.
- New forms of digital storage media packaging are driving efficiency.
Digital media enables the industry to move away from the traditional 2.5in and 3.5in form factors. While these storage devices are still very efficient and fit a proven operational model, they remain legacy approaches from the HDD era. With advancements in semiconductor NAND manufacturing, stacked packaging and board layouts, today’s digital storage media is not constrained to traditional form factors. Lower profile, lower power and better cost-optimised solutions exist for simple, entry-level use cases (such as booting) at a much better TCO. At the other end of the spectrum, we see proprietary trays leading high-density configurations with five to 10 times the capacity in the same footprint. This re-invention of storage media packaging with a wide range of options enables system architects to re-imagine data centre storage infrastructure for improved density, efficiency and reliability.
- New management models for NVMe devices are scalable and ready for prime time.
PCIe switching, which is fundamental for scaling and managing NVMe devices at the system level, can be found in every new high-performance storage deployment. It is now possible to scale NVMe with next-generation PCIe storage switches that combine the robust management features of SAS with the flexibility of modern network switches. These PCIe switches enable new storage architectures on a massive scale with the performance that is required in hyperscale data centres.
A large installed base of traditional HDD-based systems that are seeking the acceleration and TCO benefits of NVMe are being replaced by all flash arrays (AFAs) and hybrid flash arrays (HFAs). For example, a direct-attached, shared storage system – such as the open compute ‘lightning’ specification – enables high density pools of NVMe flash to address the ongoing flood of unstructured data.
Another benefit of PCIe-connected NVMe media is the ability to disaggregate and upgrade storage media separate from the other components in a system. This allows for pooling of the storage media and lower lifecycle management.
The volume server market has already enabled NVMe solutions because most SDS deployments are based on these commodity servers. New products supporting NVMe are being introduced with greater frequency from many of the world’s leading original equipment manufacturer (OEM), original design manufacturing (ODM) and joint design manufacturers (JDM).
Innovative architects
Much of the world’s unstructured data now resides in the largest, hyperscale data centres. The architects of these hyperscale data centres have become the top-down innovators who have developed (open) software stacks to store and process unstructured data while developing new applications and gaining valuable insights. They continue to drive the volume economics for flash-based media and were the first to realise the low latency benefits of NVMe. Many of their best practices are now being adopted by enterprise data centres. As the flood of unstructured data continues, storage innovators will drive system infrastructure forward to scale cloud-enabled infrastructures.
| Microsemi Headquartered in California, Microsemi has approximately 4800 employees. |
