I have heard the Netapp claim that NFS is a very nice option for ESX with performance that rivals that of FCP. I have never fully bench tested iSCSI vs NFS with ESX but I plan to do so– and I am going to test it on a Netapp device as well as a non-Netapp device (we build very nice Open-E powered arrays that support NFS and iSCSI) to see how the two protocols compare. I am interested to see how enabling link aggregation and jumbo frames will also impact each.

With the Netapp for optimal VM performance the vmdks should be block aligned as well.

Again, lots of passion here. I see the usual slam against Netapp not having ‘native’ LUNs… I say, who cares so long as it performs (ESX is presenting virtual hardware, seems to be working out just fine). NFS, even if its not as fast as FCP or iSCSI for that matter does have a nice advantage of just being able to grow the filesystem vs. adding LUN after LUN.

EMC certainly makes top performing and solid arrays. I give them the edge if a client is seeking tunable storage with ultimate performance (including flash drive support, not PAM cards).

Netapp also makes a solid array that delivers a ridiculous amount of value, especially with the new software bundles. It may not be THE fastest, but it sure is nice to manage all protocols from one OS/interface and the concept of aggregates, flexvols, and everything being double parity protected with RAID-DP is ‘fast enough’ for most. I will also say I have clients using A-SIS with ESX with much success (thankfully the newer systems have enough juice to life those painful volume size restrictions).

I look at it like this, they are both awesome products. So long as they are architected properly– they will both work well (with either protocol).

I am anxious to see how iSCSI performs vs NFS for ESX on my storagezip.com box with Open-E to have a non Netapp prospective.

peace all, I enjoyed the thread.. and don’t worry, storage is all going into the cloud soon anyway (heh…)

As far as deduplication, and I’ll keep this in general terms and vendor agnostic, primary volume deduplication is a NEW technology, and has been very slow on adoption. When you blindly merge pointers to a single set of blocks you instantly create hotspots within a disk group. If the data is striped thinly enough, you might be ok, but in a multi-reader multi-write environment which is created, this can impact commit times for the application, it can also cause misuse of read cache. Just my thoughts on the subject. Thank for reading.

-STEVEN J. Schwartz

Please accept my feedback as a token that we can stay open and frank with each other. With respect to the following:

“you’ll read exactly why I think A-SIS is a problem, is causes hot spots in applications that aren’t aware of the deduplication going on”

What customers/managers see if a FREE solution that gives you LOTS of space back. Not bad in financially strained times? If this is at the expense of applications, then this is very important. I d like to know more please.

“it forces breakage in NetApp’s own Snapshot technology”
are you referring to the fact that it can take a little bit of time to see the free space? Is this what you refer to as BREAKAGE?

“it is a fix block length deduplication which over time will see less return on active data sets, the list goes one.”

nah .-)

“I’m a Storage Technologist for a very large Vendor. ”
AHA! the cat is out. I like to get second opinions, yet it would be better for other professionals if your opinions/statements were backed up by facts. Right now this blog has got a “political” feel to it. Have you seen this:

http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html

Looking forward to hearing from you.

Eric

What exactly is a single filer pair? Are you talking about a clustered 6000 solution? 64GB RAM and 4GBNVRAM. I never considered the 6000 series a mid range array but the other user did and said he had many examples to prove the inferiority of the 6000 series against mid-range arrays? Interesting how you didn’t correct him.

Perhaps if you were a customer I’d have more time to listen to what you say. I think I’ll defer to my customers who have had other storage arrays and gone to the Netapp. After all isn’t it about what the end user finds acceptable?

Have you even bothered to read the TR on Netapp ASIS? If you did you’d realize that it clearly states that turning on ASIS across all your volumes needs to be well thought out first. Not all workloads benefit from dedupe. You’d also understand the implications on snapshots and when it best to schedule dedupe and when to schedule snapshotting. I know it’s fixed length. Variable length block length requires greater computational processing and that’s why you only see that technology on secondary storage (i.e. datadomain). The point is that it’s a FREE license that you DON”T have to turn on. There are many situations that clearly benefit from turning it on specific volumes. If you have an NFS datastore for your VM’s and turn it on you’ll see at least 50% space savings. I’ve actually done the work at many customer sites and not just theorized about it.

I don’t need to see a post to understand the differences between NFS and a block based protocol. Anyone who’s in storage clearly knows what the differences are. I guess you can shoot down all of the people who are running ESX over NFS datastores. I guess they’re all idiots. Obviously Netapp started off doing NFS and that’s their core. If you’re going to run ESX over NFS why wouldn’t you use a Netapp?

Virtualization is also my specialty, so yes I’ve followed VMware’s releases. Everybody worth their salt knows that FC was the only supported protocol when ESX first came out. VMWare has learned that IP storage is also a viable alternative if run on the right array and architected properly. ESX workloads are more focused on throughput then bandwidth. You don’t have to take my word for it but NFS compares very favourably to FC in many ESX benchmarks. If I’m a customer that doesn’t have a fabric deployed why would I want to spend the money on FC switches, HBA’s, etc as well as the added training and management of my support personnel when I can run my workloads over a technology I’ve been using for years and am very familiar with – no added switches or HBA’s, no additional training and support.

SRM is in first release. If you’ve ever attended the training or bothered to find out about the roadmap you’d notice that NFS support will be included in the next release.

I don’t think I missed the message about VI4.x. There are 3 vendors who are working with VMware tightly on their vStorage initiative – EMC (obviuosly, Equalogix, and guess who’s the third? Interesting that if Netapp is so shitty for ESX that they’d even bother working so closely with them on the next release. The trick is to move many processes away from the hypervisor and allow the storage array to do it. There are many intelligent storage arrays out there but I think that if I’m deploying ESX i’ll stick with an appliance that is going to integrate the best. There are many applications on SAN storage. I’ll stick to an appliance that can handle multiple protocols and offers a wide variety of intelligent software to help me manage my applications. If I’m worried about tiering my data that’s when I’ll have a talk with vendors like F5 who have purchased Acopia and look at tiering my data intelligently – doesn’t have to be all Netapp, I never said it had to.

I can talk to you about Hyper-V and XEN to if you’d like -which also both work great on Netapp.

Why don’t you just give me 5 points where FC is more flexible then NFS in terms of data management – you’ve peaked my interest.

Regards

Ian Forbes
Kool-Aid drinker

Thank you for trying to sell your NetApp solutions on my blog. Not exactly sure what your storage background is, but you’ve clearly drank the frame based architecture cool-aid fully. Even NetApp’s 6000 series filers bottleneck at a single filer pair, and there are several “mid-range” arrays, although I would highly doubt that NetApp would consider the 6000 series a mid-range array, and no Storage magazine would ever compare it to mid-range arrays either.

If you read some of the other posts on this blog, one, you’ll see I’m not a customer, end-user, or VAR, I’m a Storage Technologist for a very large Vendor. Secondly, you’ll read exactly why I think A-SIS is a problem, is causes hot spots in applications that aren’t aware of the deduplication going on, it forces breakage in NetApp’s own Snapshot technology, it is a fix block length deduplication which over time will see less return on active data sets, the list goes one.

NFS is a file based protocol, I have a post soon to be released about the difference between block based and file based protocols to be posted shortly. NFS has significant great use and functionality when deployed for the purposes it was designed for, shared file access.

If you’ve followed VMWare’s releases of functionality at all over the years, what you’ll see is that FC, then iSCSI, and lastly NFS is the order for all feature support. NFS is still not supported the the basic SRM feature that allows automated failover in ESX. As for your statement about VI4.x, I think you missed the message on that one. It allows VMware to be more aware of the storage, it doesn’t replace intelligent storage functionality, nor virtualized storage functionality. Last time I checked VMWare wasn’t the only application out on SAN storage either, but that is a whole other topic. Oh wait, VMWare isn’t the only OS Virtualization technology either was well.

As for FC and iSCSI offering greater flexibility, this is a topic that can’t be answered in a comment response, I’ll refer you back to the post I’ll be writing about block vs. file.

If you don’t want to turn on a free license nobody’s forcing you to. It’s called an option. I haven’t come across a customer yet who doesn’t love the fact that they see upwards of 70% space savings with this technology on their primary storage.
NFS on ESX? Please share your half dozen reasons why I’d want to go block instead. Even if you’re going to argue performance that’s certainly not the only factor people look at when deciding how to procure their datastores. Please share with us how block management is easier then file management. I just love adding extents in ESX rather then growing the volume on the Netapp and seeing it instantly translated on the ESX datastore – that really sucks.
You don’t have to pay Netapp prices to get a decent ESX datastore – true. But, if I’m looking at the total package of what I get by pairing my ESX solution with a Netapp there’s not really a comparison.

As ESX 4.0 starts to gain more traction you’re going to see a lot of functionality being moved away from the hypervisor and onto intelligent storage arrays. I have no doubt you’ll see Netapp being a primary player in that integration.

Lastly, please explain how FC and iSCSI offer greater flexibility over NFS. That one I really can’t wait to hear.

Good article Steven!

The problem? I never said there was problem with running VMWare on NFS,if you read the post subject it is a comparison between performance of NFS and iSCSI. What I said was that not all NFS and not all iSCSI is created equal. NetApp may very well have some of the fastest NFS access on the market, however, it is a marketing game for NetApp to compare NetApp’s NFS performance to MetApp’s iSCSI performance.
I was also pointing out that NetApp, in ALL instances layers ALL data on TOP of the WAFL file system.
Additionally, I was pointing to VMWare testing that shows in almost every IO block size instance that iSCSI is faster and has lower latency then NFS access. Performance of storage in a virtual environment is a MAJOR factor for customer purchases.

Thank you for your comment. Are you using deduplication too?

Regards,

Steven

I will also be posting a new blog entry specifically about NetApp’s lack of functionality in this area.

Myself, I wouldn’t be a fan of recarving a de-duped block LUN which had been formated with VMFS unless I could make VMware aware to the fact sizes had changed. Though other people could and might already be a fan.