NVMe – Flash unleashed

In this post I want to provide you with a quick introduction into NVMe, the next generation of Flash devices.

So what’s NVMe also known as Non-Volatile Memory Express or Non-Volatile Memory Host Controller Interface Specification (NVMHCI)?

The idea behind NVMe is to improve the storage stack by optimizing the way an application accesses a Flash device. NVMe offers not only a lightweight specification optimized for Flash, but also cuts some corners by removing components within the I/O path like the RAID controller. NVMe leverages PCIe as transport media which offers high bandwidth and a direct path to the hosts CPU and memory. This in turn removes another potential bottleneck, the limited bandwidth of SAS or SATA connection.

So the overall goal was to empower modern Flash devices so they can deliver their full potential and no longer being slowed down by a storage stack which was primarily designed for slow spinning disks.

I’ve found this graphic from Intel which illustrates the idea quite well.IntelNVMeSource: https://communities.intel.com/community/itpeernetwork/blog/2014/06/16/intel-ssd-p3700-series-nvme-efficiency

Below you can find a table to get a basic overview about the bandwidth PCIe can provide.

PCI Express

Per lane



2 Gbit/s

250 MB/s


4 Gbit/s

500 MB/s


7.877 Gbit/s

984.6 MB/s


15.754 Gbit/s

1969.2 MB/s

Source: http://en.wikipedia.org/wiki/PCI_Express

As you can see even a single PCIe 3.0 lane provides more bandwidth than a 6Gb/s SAS connection. The cool thing about NVMe is that it is not limited to the form factor of PCIe Flash cards, it also made its way into a new generation of 2.5” SSDs. “How?” you might ask because it’s not obvious.

Instead of putting a RAID Controller into your server to attach SSD drives via SAS or SATA as usual, you can put in a PCIe extender card instead. This adapter extends the PCIe connectivity via SFF-8639 cables to the server backplane as usual. This of course will only work with corresponding backplanes & NVMe SSDs which have a PCIe based interface like Intel’s DC P3700 series.

From what I’ve seen, a single extender plugged into a PCIe x16 slot can provide four PCIe x4 interfaces to connect four 2.5” drives. This means each SSD can make use of 31,508 Gb/s of available bandwidth! This now shifts back the bottleneck from the transport media to the device itself.

The new NVMHCI specification mentioned above defines a new host interface as well as a command set for use with non-volatile memory which can be compared to SCSI or AHCI & SATA to some extent. For example SCSI (Small Computer System Interface) also defines a protocol, a command set and interfaces like SAS. However all of them define a way how to exchange data between the attached storage device and the host’s system memory.

I’ve tried to outline the relationships below:

  • Host system <- AHCI -> HBA/Controller <- SATA -> DISK
  • Host system <- SCSI -> HBA/Controller <- SAS -> DISK
  • Host system <- NVMHCI -> FLASH DEVICE

To be able to make use of the new NVMe based devices inside your host(s) you have to have a proper driver installed which also can take care of the SCSI to NVMe translation. This enables NVMe devices to function within the existing operating system I/O stack.


Source: http://www.nvmexpress.org/wp-content/uploads/NVM-Express-SCSI-Translation-Reference-1_1-Gold.pdf

More technical details like a list of the NVM Express key attributes can be found here and news about the recently delivered NVMe 1.2 specification here.

Whereas Windows Server 2012 R2 ships with native NVMe support, ESXi was lacking a little behind. But on the 12th of November 2014 VMware has released the very first “VMware Native NVM Express Driver” for ESXi 5.5 which can be downloaded here.

Before warping up this post I also want to mention that there are also other specifications like SCSI over PCIe (SOP) or SATA Express in the making which aim to achieve the same result by leveraging existing protocols. So at some point in time it’s possible that multiple approaches co-exist, but at the moment NVMe leads the trail of innovation.

NVMe like any other next generation media, DDR4 memory or Flash storage on the memory channel for example, shows the real power of server side acceleration. Those new technologies make their way into the hypervisor relatively quickly compared to a central storage system and allows FVP customers to adapt them early on. FVP enables its users to leverage low latency & high IOPS medias as close to the virtual machine as close as possible. With the ability to easily scale performance by adding new devices or by adding new hosts to a cluster, we can scale performance as it’s required by the applications. This illustrates how future-proof FVP as storage acceleration platform really is.

Print Friendly, PDF & Email

Related Post