A Storage Technology Blog by a Technologist

Polls

Do you use a single storage protocol in your IT infrastructure?

View Results

Loading ... Loading ...

Sponsors

Categories

Archive

Blogroll

Meta

Subscribe in NewsGator Online

NFS vs. iSCSI in a VMWare environment

September 15th, 2008 by Steven Schwartz

Alien vs Predator

please also see Tortoise and the Hare? (NFS vs. iSCSI and why this Apples to Broccoli)

(to be published on 1/24/2013)

A brief update to the iSCSI vs. NFS in a VMWare environment based on 2012-2013 technologies.

 

I think the best vendor marketing campaign in recent storage history has been the full force attack of NetApp against iSCSI for use with VMWare.  Now, I can fully understand NetApp not wanting any customers using their “iSCSI”,but iSCSI in general isn’t slower than NFS for VMWare.  How do I know this?  Well common sense for one, but let’s not bring my own personal emotions and experience into this conversation.

 

I think the best part of this marketing campaign was the release of a “white paper” or “tech report” or “case study”, whatever you want to call it.  It was NetApp Technical Report TR-3697 – Performance Report: Multiprotocol Performance Test of VMWare ESX 3.5 on NetApp Storage Systems.  I think the title explains the problem with this idea from the very beginning.  The purpose of this report was the following:

 

Both NetApp storage arrays and VMware® ESX 3.5 natively support data access using three storage protocols: FC, iSCSI, and NFS. Due to the protocol flexibility and neutrality of both companies, many discussions have been had around the water cooler comparing the administrative pros and cons of the storage protocols as well as how they compare from a performance perspective in an ESX environment. This technical report, completed jointly by VMware and NetApp, shares the results of testing conducted to compare the performance of FC, software-initiated iSCSI, and NFS in an ESX 3.5 environment using NetApp storage. The results compare the performance of the three protocols with a goal of aiding customer decisions as they build out their virtual infrastructures.

 

Clear as day right?  Well what is my problem with this approach?  While NetApp does in fact support FC, iSCSI and NFS, they cannot claim to be the fastest FC, nor iSCSI block storage devices, and most likely are not the fastest NFS device on the market, however they are a NFS/CIFS device first and foremost.  Why do I point this out?  I’ll refer to an NetApp TechOnTap article.  Within this article it points out that:

Block Storage Protocols
Initial NetApp storage systems were NFS appliances. Over the years, the same core
architecture has been extended to support multiple protocols—CIFS initially, and then
block-based protocols (Fibre Channel and iSCSI). Block protocols expose LUNs, which
are special WAFL containers (“files”) that exhibit block device characteristics. They
inherit the rich lineage of WAFL—including space- and resource-efficient Snapshot
copies and clones.

So what does this mean for an end-user?  It means that when they create a FC or iSCSI volume on a NetApp filer, they are creating a very large file or “special WAFL container” that is virtualized out of the filer head as a raw block level device.  Which, of course, an OS then needs to mount through another volume manager and then place a whole other file system on that.  Clearly several layers of translation will increase latency and IO throughput.

 

So, where am I going with all of this?  Clearly, NFS is going to be one of NetApp’s best performing protocols, because they have to build everything else on top of the file level architecture.  Which leads me to my final comments.  Released in a session at VMWorld 2008 in Europe was a direct comparison of NFS vs. iSCSI as far as performance.  The title of the presentation is:

NFS & iSCSI: Performance Characterization and Best Practices in ESX 3.5

 

In this document iSCSI storage beat out NFS storage in performance, and with lower latencies.  Take a look and see for yourself.  So, when choosing a storage vendor, do some research, don’t just believe a “Tech Report“.

 

Comments always welcome….

Enhanced by Zemanta

Posted in Benchmarks, Enterprise, SAN and NAS, virtualization | 20 Comments »

  • http://storagezilla.typepad.com Storagezilla

    Well they also have some extra value add for NAS with their de-dupe option. Since the NAS head manages the FS, storage could be reallocated after the de-dup operation.

    Myself, I wouldn’t be a fan of recarving a de-duped block LUN which had been formated with VMFS unless I could make VMware aware to the fact sizes had changed. Though other people could and might already be a fan.

  • http://thesantechnologist.com Steven

    I love when the SIS functionality is brought up for NetApp. The following is NetApp’s Tech Report on Data De-duplication: (http://media.netapp.com/documents/tr-3505.pdf)

    I will also be posting a new blog entry specifically about NetApp’s lack of functionality in this area.

  • Ronny

    I don’t know what your problem is? This report compares the three different protocols ONLY on NetApp storage systems. There is no comparison with other storage vendors and it never claims that NetApp has the fastest implementation of these protocols. Besides this, I’m a happy customer using NFS for VMware on NetApp storage. cheers

  • http://www.thesantechnologist.com Steven J. Schwartz

    Ronny,

    The problem? I never said there was problem with running VMWare on NFS,if you read the post subject it is a comparison between performance of NFS and iSCSI. What I said was that not all NFS and not all iSCSI is created equal. NetApp may very well have some of the fastest NFS access on the market, however, it is a marketing game for NetApp to compare NetApp’s NFS performance to MetApp’s iSCSI performance.
    I was also pointing out that NetApp, in ALL instances layers ALL data on TOP of the WAFL file system.
    Additionally, I was pointing to VMWare testing that shows in almost every IO block size instance that iSCSI is faster and has lower latency then NFS access. Performance of storage in a virtual environment is a MAJOR factor for customer purchases.

    Thank you for your comment. Are you using deduplication too?

    Regards,

    Steven

  • http://www.storageiscool.com Not a Netapp fan…

    Though NetApp has a decent product, there are clearly other storage systems on the market today that do a much better job presenting block storage, systems that don’t have to go through several layers of translation just to serve up block-level storage. I have baked off several mid-market arrays against NetApp’s block LUN’s, and in ALL instances NetApp had to put almost twice as much disk behind their controller to keep up (yes, even the 6000 series with all the cache), not to mention all the disk overhead associated with Data OnTap, WAFL, and snapshot reserves. I love baking off against them. The de-dupe (A-SIS) isn’t even close to what it is marketed as; there are so many caveats that I can’t see how anyone would go through the trouble. I suppose it would help reclaim some of the space eaten up the the OS and file system. NFS on ESX? There are at least half a dozen reasons to go with block instead of NFS…NetApp does optimize their NFS in a couple ways, but at the end of the day you don’t have to pay NetApp prices to get an excellent ESX datastore; iSCSI and FC will do a much better job with alot more flexibility!

    Good article Steven!

  • Ian Forbes

    What mid market arrays have you baked off against the 6000 series? I’d be interested to know. Can you expand what you mean about the ASIS not being marketed to be what it is? It’s a space saving solution. What caveats are you talking about? What trouble? It’s a free license that takes all of one command to turn on. What trouble are you referring to exactly? Why wouldn’t you want to have a free solution to offer you greater space savings on your primary storage?? You want to purchase more disk? I work for a reseller and love customers like you. Helps me make my number :)

    If you don’t want to turn on a free license nobody’s forcing you to. It’s called an option. I haven’t come across a customer yet who doesn’t love the fact that they see upwards of 70% space savings with this technology on their primary storage.
    NFS on ESX? Please share your half dozen reasons why I’d want to go block instead. Even if you’re going to argue performance that’s certainly not the only factor people look at when deciding how to procure their datastores. Please share with us how block management is easier then file management. I just love adding extents in ESX rather then growing the volume on the Netapp and seeing it instantly translated on the ESX datastore – that really sucks.
    You don’t have to pay Netapp prices to get a decent ESX datastore – true. But, if I’m looking at the total package of what I get by pairing my ESX solution with a Netapp there’s not really a comparison.

    As ESX 4.0 starts to gain more traction you’re going to see a lot of functionality being moved away from the hypervisor and onto intelligent storage arrays. I have no doubt you’ll see Netapp being a primary player in that integration.

    Lastly, please explain how FC and iSCSI offer greater flexibility over NFS. That one I really can’t wait to hear.

  • http://www.thesantechnologist.com Steven J. Schwartz

    Ian,

    Thank you for trying to sell your NetApp solutions on my blog. Not exactly sure what your storage background is, but you’ve clearly drank the frame based architecture cool-aid fully. Even NetApp’s 6000 series filers bottleneck at a single filer pair, and there are several “mid-range” arrays, although I would highly doubt that NetApp would consider the 6000 series a mid-range array, and no Storage magazine would ever compare it to mid-range arrays either.

    If you read some of the other posts on this blog, one, you’ll see I’m not a customer, end-user, or VAR, I’m a Storage Technologist for a very large Vendor. Secondly, you’ll read exactly why I think A-SIS is a problem, is causes hot spots in applications that aren’t aware of the deduplication going on, it forces breakage in NetApp’s own Snapshot technology, it is a fix block length deduplication which over time will see less return on active data sets, the list goes one.

    NFS is a file based protocol, I have a post soon to be released about the difference between block based and file based protocols to be posted shortly. NFS has significant great use and functionality when deployed for the purposes it was designed for, shared file access.

    If you’ve followed VMWare’s releases of functionality at all over the years, what you’ll see is that FC, then iSCSI, and lastly NFS is the order for all feature support. NFS is still not supported the the basic SRM feature that allows automated failover in ESX. As for your statement about VI4.x, I think you missed the message on that one. It allows VMware to be more aware of the storage, it doesn’t replace intelligent storage functionality, nor virtualized storage functionality. Last time I checked VMWare wasn’t the only application out on SAN storage either, but that is a whole other topic. Oh wait, VMWare isn’t the only OS Virtualization technology either was well.

    As for FC and iSCSI offering greater flexibility, this is a topic that can’t be answered in a comment response, I’ll refer you back to the post I’ll be writing about block vs. file.

  • Ian Forbes

    I wasn’t trying to sell anything. I’m a storage architect and we partner with many storage vendors – Hitachi, IBM, Netapp, HP etc. Everyone has their bias and mine is Netapp. If you read my comments carefully you would see I was merely asking for clarification on the other person’s comments. Interesting that you didn’t respond to his comments. Do I drink the Netapp kool-aid? Not really but I certainly understand the technology. You’re a Netapp basher so there’s really no use arguing with you. It doesn’t matter what I say you’ll construe my comments as to how you see fit. It’s a religous arguement and I’ll accept that we agree to disagree.

    What exactly is a single filer pair? Are you talking about a clustered 6000 solution? 64GB RAM and 4GBNVRAM. I never considered the 6000 series a mid range array but the other user did and said he had many examples to prove the inferiority of the 6000 series against mid-range arrays? Interesting how you didn’t correct him.

    Perhaps if you were a customer I’d have more time to listen to what you say. I think I’ll defer to my customers who have had other storage arrays and gone to the Netapp. After all isn’t it about what the end user finds acceptable?

    Have you even bothered to read the TR on Netapp ASIS? If you did you’d realize that it clearly states that turning on ASIS across all your volumes needs to be well thought out first. Not all workloads benefit from dedupe. You’d also understand the implications on snapshots and when it best to schedule dedupe and when to schedule snapshotting. I know it’s fixed length. Variable length block length requires greater computational processing and that’s why you only see that technology on secondary storage (i.e. datadomain). The point is that it’s a FREE license that you DON”T have to turn on. There are many situations that clearly benefit from turning it on specific volumes. If you have an NFS datastore for your VM’s and turn it on you’ll see at least 50% space savings. I’ve actually done the work at many customer sites and not just theorized about it.

    I don’t need to see a post to understand the differences between NFS and a block based protocol. Anyone who’s in storage clearly knows what the differences are. I guess you can shoot down all of the people who are running ESX over NFS datastores. I guess they’re all idiots. Obviously Netapp started off doing NFS and that’s their core. If you’re going to run ESX over NFS why wouldn’t you use a Netapp?

    Virtualization is also my specialty, so yes I’ve followed VMware’s releases. Everybody worth their salt knows that FC was the only supported protocol when ESX first came out. VMWare has learned that IP storage is also a viable alternative if run on the right array and architected properly. ESX workloads are more focused on throughput then bandwidth. You don’t have to take my word for it but NFS compares very favourably to FC in many ESX benchmarks. If I’m a customer that doesn’t have a fabric deployed why would I want to spend the money on FC switches, HBA’s, etc as well as the added training and management of my support personnel when I can run my workloads over a technology I’ve been using for years and am very familiar with – no added switches or HBA’s, no additional training and support.

    SRM is in first release. If you’ve ever attended the training or bothered to find out about the roadmap you’d notice that NFS support will be included in the next release.

    I don’t think I missed the message about VI4.x. There are 3 vendors who are working with VMware tightly on their vStorage initiative – EMC (obviuosly, Equalogix, and guess who’s the third? Interesting that if Netapp is so shitty for ESX that they’d even bother working so closely with them on the next release. The trick is to move many processes away from the hypervisor and allow the storage array to do it. There are many intelligent storage arrays out there but I think that if I’m deploying ESX i’ll stick with an appliance that is going to integrate the best. There are many applications on SAN storage. I’ll stick to an appliance that can handle multiple protocols and offers a wide variety of intelligent software to help me manage my applications. If I’m worried about tiering my data that’s when I’ll have a talk with vendors like F5 who have purchased Acopia and look at tiering my data intelligently – doesn’t have to be all Netapp, I never said it had to.

    I can talk to you about Hyper-V and XEN to if you’d like -which also both work great on Netapp.

    Why don’t you just give me 5 points where FC is more flexible then NFS in terms of data management – you’ve peaked my interest.

    Regards

    Ian Forbes
    Kool-Aid drinker

  • eric bar

    Dear Stephen,

    Please accept my feedback as a token that we can stay open and frank with each other. With respect to the following:

    “you’ll read exactly why I think A-SIS is a problem, is causes hot spots in applications that aren’t aware of the deduplication going on”

    What customers/managers see if a FREE solution that gives you LOTS of space back. Not bad in financially strained times? If this is at the expense of applications, then this is very important. I d like to know more please.

    “it forces breakage in NetApp’s own Snapshot technology”
    are you referring to the fact that it can take a little bit of time to see the free space? Is this what you refer to as BREAKAGE?

    “it is a fix block length deduplication which over time will see less return on active data sets, the list goes one.”

    nah .-)

    “I’m a Storage Technologist for a very large Vendor. ”
    AHA! the cat is out. I like to get second opinions, yet it would be better for other professionals if your opinions/statements were backed up by facts. Right now this blog has got a “political” feel to it. Have you seen this:

    http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html

    Looking forward to hearing from you.

    Eric

  • http://www.thesantechnologist.com Steven J. Schwartz

    Firstly, it is Steven. So if you’ve read the about me closely enough, you’d see that yes I work for Dell, and in General, accept for the occasional blog war with a competitor, I’m pretty fair. Political? I’ve been called pretty much everything in the book, both publicly and privately, but never political.

    As far as deduplication, and I’ll keep this in general terms and vendor agnostic, primary volume deduplication is a NEW technology, and has been very slow on adoption. When you blindly merge pointers to a single set of blocks you instantly create hotspots within a disk group. If the data is striped thinly enough, you might be ok, but in a multi-reader multi-write environment which is created, this can impact commit times for the application, it can also cause misuse of read cache. Just my thoughts on the subject. Thank for reading.

    -STEVEN J. Schwartz

  • http://www.storagezip.com independent storage guy…

    interesting thread, I love the passion :-) Well folks, here is my two cents after selling EMC, Netapp, and an array of others (pun intended) for the past 11yrs.

    I have heard the Netapp claim that NFS is a very nice option for ESX with performance that rivals that of FCP. I have never fully bench tested iSCSI vs NFS with ESX but I plan to do so– and I am going to test it on a Netapp device as well as a non-Netapp device (we build very nice Open-E powered arrays that support NFS and iSCSI) to see how the two protocols compare. I am interested to see how enabling link aggregation and jumbo frames will also impact each.

    With the Netapp for optimal VM performance the vmdks should be block aligned as well.

    Again, lots of passion here. I see the usual slam against Netapp not having ‘native’ LUNs… I say, who cares so long as it performs (ESX is presenting virtual hardware, seems to be working out just fine). NFS, even if its not as fast as FCP or iSCSI for that matter does have a nice advantage of just being able to grow the filesystem vs. adding LUN after LUN.

    EMC certainly makes top performing and solid arrays. I give them the edge if a client is seeking tunable storage with ultimate performance (including flash drive support, not PAM cards).

    Netapp also makes a solid array that delivers a ridiculous amount of value, especially with the new software bundles. It may not be THE fastest, but it sure is nice to manage all protocols from one OS/interface and the concept of aggregates, flexvols, and everything being double parity protected with RAID-DP is ‘fast enough’ for most. I will also say I have clients using A-SIS with ESX with much success (thankfully the newer systems have enough juice to life those painful volume size restrictions).

    I look at it like this, they are both awesome products. So long as they are architected properly– they will both work well (with either protocol).

    I am anxious to see how iSCSI performs vs NFS for ESX on my storagezip.com box with Open-E to have a non Netapp prospective.

    peace all, I enjoyed the thread.. and don’t worry, storage is all going into the cloud soon anyway :-) (heh…)

  • Pingback: Performance stats for ESXi VMs on Openfiler NFS

  • Please answer Ian

    I know this is an old post and you may not like the tone of Ian Forbes’s last comment but I would be very interested in hearing your thoughts on his questions. You seem to have ignored them all. Thank you.

  • http://www.thesantechnologist.com Steven J. Schwartz

    I’ve been asked to reply to the questions in this comment, so I’ll do my best to capture them and respond.

    Let me first start by saying I’m not a NetApp “basher”. I have always believed NetApp to have a good technology, and many years ago when they started they addressed an open gap in the network attached storage space. When this post was first written, 1GigE was the primary NFS and iSCSI connection method, and FC was deep into 4Gb architectures. NFS was barely supported by VMWare (always behind on supporting the latest features on NFS attached storage), and VMWare was really just starting to get into complex storage requirements. This is far from the case in the middle of 2011. We are almost 3 years past this post, and I’m working on creating an updated version of this, which paints NFS storage in a much better light, makes FC seem like a dinosaur, and iSCSI right along with it. However, I will always say, just like I said with FC, iSCSI and NFS in the past, not all vendors are created equal, and in general, people can make very good protocol decisions, while still choosing a very poor performing storage product.

    I single filer pair as I referred to it in this post was specifically talking about the single active/passive-passive/active architecture of NetApp.

    In the world of network attached storage I would still stand by the statement that NetApps 6000 series filers are a mid-range product, there are far superior products that address the high-end, however, if I address this fairly, Gartner considers any NAS solution over $100k a high-end product, and in this case, most NetApp 3000 series and 6000 series would be lumped into that category.

    End users may find many things acceptable, I know some that run Netgear NAS devices with no RAID protect and are quite happy, however, if my customer cares about the data, then I’m going to make sure they have a solution that meets the business requirement of the company, not an end-user’s idea of quality.

    Of the customer I know running NetApp with A-SIS in a VMWare environment, at best they have seen 30% space savings, over what the NetApp filer was using prior to enabling the A-SIS feature. They also have gotten into non-recoverable situations because of the forced elimination of snapshots in order to run and save the space. There is also a huge overhead for running A-SIS, but I’ve mentioned this before.

    Why wouldn’t I use NetApp for VMWare NFS storage? In a small shop and small deployment I might. The 15TB file system size limitation is one of the huge reasons I wouldn’t use it, if I have to manage a bunch of file systems, then I might as well just run anything I want. The overhead of RAID-DP is another reason I wouldn’t, both in terms of performance and capacity. The problems with running the WAFL file system at anything over 65% utilization is another reason I would shy away from NetApp. The licensing and support model would be another reason. Lastly, the complete lack of scalability for when I do need more capacity or performance and now have to manage disparate NetApp clusters.

    Um, I don’t know of a VMWare storage benchmark, so not sure how anybody compares using one. I will agree that NFS is much easier to manage for datastores then any block level storage device.

    SRM while supported, really relies on the vendor having a SRM adapter, so regardless of “VMWare” supporting a protocol, if the vendor doesn’t, it really doesn’t matter.

    VAAI is really the next big thing, and NFS support is in the works.

    All this said, I never in my post said FC is the better protocol, I said iSCSI was better than FC, and in this year (2011) I think NFS is the best protocol.

  • Please answer Ian

    Steve,
    Thanks a bunch for addressing those comments and for all the quality information you have provided here. I really look forward to your updated version of this article. Keep up the good work!

  • Pingback: Performance stats for ESXi VMs on Openfiler NFS - Admins Goodies

  • Pingback: Xen Live Migration Across Network Storage - Admins Goodies

  • Sean

    Although prior to 8.1, the active/passive cluster architecture was a concern for NetApp, 8.1 changes that so that you can have multiple controllers accessing the same disks. This is a active-active-active-… architecture. There is likely some limitations to this, as only two controllers have physical connections to the disk shelves.

    I discovered this thread when I was searching for VMWare deployments with NFS. I am not a SAN guy, but have has 14+ years with NetApp and NAS architectures, and this is why I am more comfortable going this route. My IT department, which will have no part in managing my solution, are SAN guys. They deployed their VMWare environments using iSCSI and FC, and are constantly bitching and complaining, and making poor excuses why their environment sucks, yet they won’t take actions on the vendor’s recommendations. This is ok, as it has actually driven my VMWare deployment as a replacement for theirs, with the goal of managing our network.

    Ultimately, I think I would agree that LUN support on the NetApp is probably not the best, but I have always hated the traditional volumes, where you have to assign LUNs to specific disks, and end up having such poor flexibility in the size of volumes, or having to purchase way more disk than you need just because you need to provide SAN access to multiple machines.

    I am glad to hear that NFS is a viable alternative to being forced to use LUN via iSCSI or FC.

  • http://www.thesantechnologist.com Steven J. Schwartz

    Sean,

    There are some very big mis-conceptions about NetApp cluster mode and NetApp’s traditional filer mode, as I’ll refer to it. The Cluster Mode is an implementation of SpinFS and is a clustered file system which has different performance and use cases, implements a segmented file system model. Traditional NetApp 2 node clusters still are active-passive passive-active for a given file system, and don’t implement a distributed lock manager.

  • Nate Stuyvesant

    Outside of vendor considerations, one thing I’m curious about is whether NFS or iSCSI are better specifically for vCenter Lab Manager and View. These products provide the capability to produce linked virtual machine clones (essentially a delta disk). But a limitation was that the linked clones had to exist on the same Datastore. As a result, a Datastore with a single iSCSI-based LUN could become very densely packed with running virtual machines. The result would be very poor performance. I’m curious as to whether NFS might be a better fit for this type of environment. I would think we’d be seeing more of this since vSphere 5 includes Fast Provisioning which is the successor to the linked cloning approach.