/ PLN
You've got fast NVMe drives in your servers, but each one is tied to a single machine – and when one is running out of space while another has plenty to spare, you can't move it across. NVMe-oF (NVMe over Fabrics) solves exactly this problem: it lets a server access NVMe drives over a network as if they were sitting locally – with latency measured in microseconds, not milliseconds.
In short, it's a way to decouple fast flash storage from a single server and share it across the whole infrastructure. Below, we explain how it works, how it differs from regular NVMe, which network variants come into play, and what it actually costs – without glossing over the downsides.
What does NVMe-oF actually mean?
Let's start with what surprises most people: NVMe-oF isn't a new protocol. It's an extension of the same NVMe you already know from SSDs inside a server, applied to an external network – called a fabric. The standards body (NVM Express) defines it plainly as NVMe running over a transport other than PCI Express.
How much "the same" are we talking about? Around 90% of the NVMe over Fabrics protocol is identical to local NVMe – the same namespaces, I/O and administrative commands, event handling. The difference comes down to how commands travel across the network instead of the PCIe bus. As a result, the operating system sees a remote drive almost the same way it sees a local one, which simplifies both deployment and maintenance.
Why is this faster than old-style networked storage?
This is the crux of the whole thing. Classic networked storage over iSCSI typically has latency 4–5 times higher than direct access to local memory. For a database or virtualization environment, that's the difference between "runs smoothly" and "noticeably stutters." NVMe-oF was designed specifically to close that gap.
The scale of improvement is striking. The design goal states that the fabric network itself should add no more than 10 µs of latency. Early implementations achieved around 100 µs as experienced by the host, and the next generation brought that down to 20–25 µs. Below you can see this progress alongside the starting point, iSCSI:
Latency drops by roughly an order of magnitude compared to classic iSCSI. Indicative values, based on NVM Express data.
What is this actually useful for?
The biggest benefit is organizational, not just technical. Many NVMe deployments relied on drives connected via PCIe to a single server, leaving expensive, fast flash trapped in individual machines and underutilized – the industry has a term for this: "stranded flash." NVMe-oF lets you pool these resources together, shared by multiple servers at once.
Where we see this most often:
- shared all-flash arrays – fast storage as a shared resource for multiple hosts, instead of isolated islands in separate servers,
- virtualization and databases – where low write latency and predictable performance matter,
- HPC and AI environments – where storage throughput can become the bottleneck for training and computation.
These benefits aren't just theoretical. In HPC benchmarks, an adaptive NVMe-oF implementation achieved up to 7 times the throughput of a classic networked NFS file system. And the standard's scalability reaches far – a single NVMe subsystem can expose tens of thousands of ports, making it a foundation for truly large, shared environments. If you're planning storage like this, look into all-flash arrays or Dell PowerStore, and for the media itself – NVMe drives.
Three paths to NVMe-oF: TCP, Fibre Channel, and RoCE
NVMe-oF isn't tied to a single networking technology – and that's a major advantage. You have several transports to choose from, each with a different trade-off between performance, cost, and how much deployment effort it takes.
NVMe/TCP runs on standard Ethernet switches with no special configuration – the lowest barrier to entry. The price for that simplicity is higher latency and greater CPU usage, since there's no hardware offload. The standard was actually created specifically so you could use your existing network, in cases where older cards and switches don't support RDMA.
FC-NVMe (over Fibre Channel) is the choice when you already have a SAN network. With 16GFC infrastructure or faster, you don't need to replace switches – a firmware update to your HBA cards is enough, and a single adapter can handle both SCSI and NVMe traffic simultaneously. The newer version of the standard handles error recovery up to 30 times faster than previous methods.
RoCE (RDMA over Ethernet) delivers the highest performance, but comes with a catch – it requires complex configuration of every switch along the path (a lossless network), RDMA-capable cards, and administrators who know how to manage it. This shows up clearly in concurrency figures compared to TCP:
InfiniBand (RDMA) and RoCE make better use of concurrency than TCP under full load. Data: NSF research.
|
Transport |
Performance |
Deployment |
Who it's for |
|
NVMe/TCP |
good, higher latency and CPU usage |
simplest – standard Ethernet |
starting out without replacing the network |
|
FC-NVMe |
high, hardware offloads |
easy with existing SAN (16GFC+) |
companies with a Fibre Channel network |
|
RoCE (RDMA) |
highest, lowest latency |
complex – lossless network, RDMA expertise |
single rack, maximum performance |
Does NVMe-oF cost more than regular NVMe?
The honest answer: it depends on the transport, and yes – the RDMA variant can cost noticeably more. In a direct comparison of 40 Gb/s cards, a RoCE card cost $420, versus $760 for iWARP – both markedly more expensive than standard Ethernet cards. Switches for RoCE are also pricier, which is why most RDMA deployments stay confined to a single rack, where the short distance maximizes the benefit of low latency.
There's also a hidden cost to factor in – operational complexity. RoCE relies on a flow-control mechanism (PFC) that, in data center networks, can lead to deadlocks and "storms" of pause frames. Even the largest operators have run into this as a source of outages: in one documented case, packet loss on a RoCE network during AI training resulted from a combination of flawed buffer assumptions, an aggressive firmware update, and a sudden traffic spike. The takeaway is simple – the "fastest" transport can also be the most troublesome, which is why NVMe/TCP and FC-NVMe are a more sensible starting point for many companies.
If you're building this from the hardware side, we can select the right network cards, switches, and optical modules for your chosen transport – from plain Ethernet to Fibre Channel.
Which variant should you choose?
The rule we apply most often is surprisingly simple. Already have a SAN network? FC-NVMe gets you fast storage with almost no infrastructure changes. Want to start on a small budget? NVMe/TCP runs on standard Ethernet. Need maximum performance within a single rack and have the expertise? Then RoCE – with the understanding that it comes with higher cost and complexity.
Choosing the right transport, array, and media depends on what actually needs to run on your infrastructure. Tell us what workload you're planning – virtualization, databases, AI – and we'll select an all-flash array, NVMe drives, and network to match that scenario. Every configuration you receive is tested, ready to deploy, and covered by warranty. For ready-made systems, check out Dell PowerVault arrays and PowerStore, and for compute – our AI servers.
FAQ
What's the difference between NVMe-oF and regular NVMe?
NVMe-oF is the same NVMe, just running over an external network (fabric) instead of the local PCIe bus. About 90% of the protocol is identical – the main difference is how commands are transported, which lets a server access remote drives almost as if they were local.
What transports does NVMe-oF support?
Three main ones: NVMe/TCP (standard Ethernet, simplest to deploy), FC-NVMe (Fibre Channel, good if you already have a SAN), and RoCE (RDMA over Ethernet, highest performance at the cost of complexity).
Is NVMe-oF faster than iSCSI?
Yes, significantly. Classic iSCSI typically has latency 4–5 times higher than local access, while newer NVMe-oF deployments get down to around 20–25 µs. That's a real difference for databases and virtualization.
Does NVMe-oF cost more than regular NVMe?
Depends on the transport. NVMe/TCP runs on your existing network, so the cost is low. RDMA variants (RoCE) require more expensive cards and switches, plus more administrator expertise – that's where the cost climbs.
When should you choose FC-NVMe over RoCE?
When you already have a Fibre Channel network (16GFC or faster) – you can deploy FC-NVMe without replacing switches, and a single adapter handles both SCSI and NVMe traffic. RoCE makes sense when you need maximum performance within a single rack.
What is NVMe-oF best suited for?
Shared all-flash arrays, virtualization, latency-sensitive databases, and HPC/AI environments. It solves the problem of flash storage being "trapped" in individual servers by pooling it into a shared resource.
Sources
- NVM Express, Inc. – NVM Express over Fabrics Revision 1.1a – https://nvmexpress.org/wp-content/uploads/NVMe-over-Fabrics-1.1a-2021.07.12-Ratified.pdf
- NVM Express, Inc. – NVMe over Fabrics Overview – https://nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
- NVM Express, Inc. – NVM Express Base Specification, Revision 2.3 – https://nvmexpress.org/wp-content/uploads/NVM-Express-Base-Specification-Revision-2.3-2025.08.01-Ratified.pdf
- SNIA – Let's Talk 'Fabrics' (J Metz) – https://www.snia.org/sites/default/files/ESF/Lets-Talk-Fabrics-NVMe-Over-Fabrics.pdf
- Patent USA – NVM Express Controller for Remote Access over Ethernet-Type Networks – https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/10503679
- Patent USA – Storage Transactions with Predictable Latency – https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/12153962
- UNH-IOL – NVMe: Lower Prices, More Features Expand Use Cases – https://www.iol.unh.edu/news/2019/06/04/nvme-lower-prices-more-features-expand-use-cases
- Kashyap, Lu – NVMe-oAF: Towards Adaptive NVMe-oF (HPDC '22) – https://par.nsf.gov/servlets/purl/10350403
- Kashyap et al. – Impact of Commodity Networks on Storage Disaggregation with NVMe-oF (NSF) – https://par.nsf.gov/servlets/purl/10299229
- Marvell – FC-NVMe (NVMe over Fibre Channel) White Paper – https://www.marvell.com/content/dam/marvell/en/public-collateral/fibre-channel/marvell-fibre-channel-nvme-over-fabrics-white-paper.pdf
- Mittal et al. – Revisiting Network Support for RDMA (UIUC) – https://radhikam.web.illinois.edu/irn.pdf
- Meta / Stanford – RDMA over Ethernet for Distributed AI Training at Meta Scale (SIGCOMM) – https://cs.stanford.edu/~keithw/sigcomm2024/sigcomm24-final246-acmpaginated.pdf
