An Introduction to Non-Volatile Memory Express (NVMe)

Before the advent of flash the storage world was stuck in a bit of a rut. Sure, hard disk drives were growing larger in capacity and getting physically smaller, but not much else was changing. Then along came flash. Flash redefined the way we thought about storage performance and capacity. It brought to the mainstream features such as data compression and deduplication. It did so in a manner that didn’t cause a workload crushing performance impact. All types of new changes happened in the storage world. Laptops booted faster; enterprise storage arrays got faster, and storage workloads began to consolidate – but something was still wrong. All that happened was the slow spinning media was being replaced by a really fast flash drive. The flash storage was retrofitted into existing mediums and protocols, which were still a serious bottleneck. We’re driving down an 8 lane flash highway, but 6 of the lanes are closed. In order to unlock the full potential of flash storage a new way to access the data stored on the drives needed to be designed. No longer could we rely on SCSI, AHCI, SAS, and SATA. Instead a new standard, NVMe, needed to be created.


NVMe, or NVM Express, stands for Non-Volatile Memory Express. As the name implies NVMe was built from the ground up to work with high performance non-volatile memory technologies. NVM Express is a specification which standardizes the accessing of high performance flash storage through a PCIe bus. This encompasses the host driver as well as the physical form factor the flash storage uses. Prior to NVM Express several flash storage vendors were designing devices which were making use of PCIe inside a computer. This made sense as PCIe offered many benefits over SAS and SATA connections, the primary of which is sheer speed followed closely by PCIe’s physical proximity to RAM and the processor. The problem was each company was designing their own interface and nothing was standard about it. It took the formation of a working group of companies –many of whom are competitors — coming together to develop an open standard for everyone to use.

Today the protocol for controlling nearly every storage device is either SAS or SATA, both of which have severe limitations in the number of operations it can process at a given time. An operation, called a command, which is not immediately run is placed in a queue to be completed later. A single device can only have one queue and it may only contain a small number of commands. This is known as its queue depth. If the queue is full it must wait to be placed in the queue which causes a large impact on performance. For example SATA has a single queue with a maximum of 32 commands. On the other hand we have NVM Express which can have over 65,000 individual queues and each queue can have 65,000 commands in it. This is one of the keys that allow multi-core CPUs to truly push the limits of flash storage. Flash is an inherently parallel storage medium, but without NVMe it is treated as serial. NVMe is architected from the ground up to take advantage of the design of flash storage.

NVMe also allows for a physical flash device to be partitioned into multiple logical storage devices called namespaces. Each namespace can have a unique pathway or pathways between the CPU and the storage device. This allows modern applications to extend parallelism down to the storage device itself. A single IO pathway can exist for a core, a process, or even a single thread without the need for locking. In addition the number of commands needed to actually access data using NVMe is far less than with SCSI. The biggest hurdle for NVM Express is that SCSI is such a widely adopted protocol its nearly impossible to find a device which does not support it. That is why it was so important for a broad vendor neutral group to come together to develop the standard. This has led to many operation systems, including Linux and Windows, having NVMe support.In order to unlock the full potential of flash storage the industry has come together to build a new standard. That standard is currently being improved to make the best use of the resources flash storage has to offer. Today NVMe is mostly a standard for an individual computer to access local storage; however that is changing. Standards are currently being worked on which will allow for NVM Express over various types of networks including Ethernet and Fiber Channel. When this happens enterprise All Flash Arrays will push speeds to the next level. In the meantime flash array vendors are working in NVMe interconnects for inside the array, with at least one product already on the market. It is not often the storage industry comes together to embrace something the way they are with NVMe. The future for storage interface is surly changing for the better with NVMe.

Leave a Reply

Your email address will not be published. Required fields are marked *