Image via Wikipedia

     Time for an update about storage protocols, I say protocols, because the transport layer is something that is in flux right now.  NFS, iSCSI, Fibre Channel, and FCOE (Fibre Channel over Ethernet) are storage PROTOCOLS!  It is pretty important to understand this as base knowledge when discussing storage connectivity to a VMWare environment.  So what does this mean for you.

     Recent published White Paper from VMWare shows in little detail the testing they did for these protocols.  However, this assumes that ALL storage and storage solutions are equal, which we all know NOT to be the case.  The white paper can be found here.  So let me first explore what they tested, and then we can look at some changes to the components.

The Storage:

     So I have when a software company does things like this, they take a “storage server” and emulate different storage protocols in order to test them.  This configuration used 9 disks, how it was configured, what type of “storage server” wasn’t disclosed, and how the LUNs were emulated wasn’t disclosed either.  So I can’t really make too many judgments, because they will end up being assumptions and we all know what you get from that.  I can however say that this was not “purpose built” and seemed to have been thrown together for this test, most likely using some Linux distribution.  The problem with this approach is it doesn’t take into account the massive development that a storage vendor has gone through either with hardware or software to get the most out of a protocol.

The performance:

     Looking just at the basic configuration, comparing GigE ethernet (with any protocol) compared to 4Gb FC doesn’t seem like a fare test.  Looking closely, VMware really was trying to prove out scalability regardless of storage protocol.  They also pointed out clearly that not using the VMFS might have a significant impact on performance, but they excluded it in order to not give a block level device an advantage over NFS mounted storage.  They also clearly pointed out that an update to this White paper was needed in order to consider 8Gb FC and 10GigE, which I think will be a much closer comparison of technologies.  If you look at the differences in performance, 4Gb FC was almost exactly 4x the performance of any of the 1Gb Ethernet used in the test, which I would expect.  It also showed that the packet size/block size was a significant differentiator because of the extra processing that needs to occur with a 1500MTU packet limitation on GigE.

The Point:

     The point is simple, why even bother putting out a “White Paper” that really proves nothing, well, it does prove something.  When using basic volumes iSCSI and NFS have pretty much equal performance over a single GigE connection.  NFS has more of a CPU overhead (minimally) then software initiator based iSCSI, and if you are counting clock cycles, then you really need to go with iSCSI or FC HBAs.  I would have rather seen NFS compared to VMFS over the iSCSI connections, that would have at least compared similar access functionality.

      So changes I would like to see, compare 1Gb FC to GigE for these protocols.  Use the best of breed NAS, iSCSI, and FC arrays for these tests (oh that would create a field day for us out here in the stoblogosphere).  Compare like functionality, NFS offers shared access, VMFS offers shared access.  In any case, someone has to write something over there, I would have named this What Paper, “VMWare 4.0 Scales Regardless of Storage Protocol”.

Comments always welcome.

Image by wynner3 via Flickr

About a week ago I tweeted that I had been invited to a storage vendor’s event, was invited as a potential customer by two different parties.  Funny thing about email lists, people get invited that you wouldn’t expect to attend.  I took off my storage hat, put on my analyst hat and promised a few old friends that I would behave.  I’ve competed against Compellent in the past (sometimes very aggressively!).  I’ve also said some nice things about the technology as well.

Compellent

Strong storage player, public company surviving in a harsh IT economy, and locally here in Colorado working with some great internal talent and one of my favorite partners.  Basic premise of the storage technology is controller based disk array (controllers are commodity based architecture which I’ve mentioned before) with a plethora of storage software services.  The special part that they bring to the table is at the storage service level, Data Progression™, Fast Track, replication, etc.  They have a very compelling story, proven scalability and a growing revenue and customer base.  The recent introduction of SSD support brings what most like to consider as a Tier0 storage layer, and although they have SATA, I would really like to see a strong archival tier in the near future.

Lewan and Associates

Lewan, is one of my favorite local integrators.  They did a very nice presentation about how they have several lines of business, specifically broken into two main lines of business, the traditional printer/copier side, and the IT Solutions side.  Lewan was relatively recently acquired by Xerox, but remains a very independent subsidiary.  What I liked the most about Lewan’s presentation was that they get the whole IT picture, and to date have had some of the key technology partners to create best-of-breed solutions for customers in the Rocky Mountain region.  The quick elevator pitch from Lewan?  D.N.A. – Data Center, Network, and Application Delivery.  They cover it all, and do it not only from a pure product basis, but from a consultative model that keeps the customers coming back!

Latisys

Latisys was another sponsor of this event, and they have some great solutions to the remote data center/hosting/co-location problem.  The purpose of the presentation on this day was to talk about Compellent’s place within Latisys, specifically to offer a target for secure remote replication.  This is something that any customer should look at that is currently running a Compellent SAN, because it reduces the complexity and I would assume costs associated in owning your own remote installation of Compellent gear, while giving you a place to recover too.  The better play they are offering however, is the ability to bring up your VMware based environment in a disaster, which is something that is very powerful for business continuance.

The SAN Technologist

I felt like a spy, I got funny looks, got directly asked if I had any questions, but I behaved and while I made no promises, I learned, I liked the things I saw, and I could without loosing sleep write something that is meaningful about these companies.  It isn’t everyday that you get a chance to see old friends, not try to destroy them, and leave with friendship renewed.  Quick shout out to my lovely mother who will greatly enjoy the Wizard of Oz reference, although not Judy’s greatest role, it is a good one. 

Image via Wikipedia

     Couldn’t make it to VMWorld this year for various reasons, but I have been eagerly listening in to all the chatter, social networking, and general fun being spewed across the internet.  There have been funny cartoons, funny videos, blogs, tweets, Facebook updates, etc.  The updates and news coverage is pretty great, but there has been one vendor’s daily updates that of course are self serving, which they should be, but are a brag about something 100% in their control.

     Not trying to launch a war here, but twitter man @HPStorageGuy, (who you should follow because he has some good thoughts when not bragging about LeftHand’s VSA)  Calvin Zito has been busy tweeting these past few days.  Typically he talks up the LeftHand Networks, now owned by HP, iSCSI storage solutions and other HP storage products.  Recently, Calvin has been bragging about how HP has the most VMWare HCL certified storage systems, server systems, and the ONLY virtual storage appliance that is certified.  Not sure everyone knows, but getting on VMWare’s HCL is an easy, but expensive task.  So, considering that all it takes is money and a little time, why brag about something like this? Regardless, Congratulations!!!  Also, Congratulations to RedHat for having the most certified OSs!!!

Image by Picture Perfect Pose via Flickr

     Nothing more to say that hasn’t been said already.  Basically, if you’ve have VMWare running on NetApp appliances ( SPECIFICALLY USING NFS), and have followed the best practices guides dated prior to Sept. 2008, then your IT environment might be at risk.

     Please contact NetApp ASAP for the configuration changes that should be made.  I am also including the information posted by the Storage Monkeys:

There are a couple of important things to note:

• This is not a NetApp problem, but rather the result of a recommendation from NetApp. At that time, it was believed that this setting was necessary.
• In most cases VMs affected by this split-brain scenario will get corrupted and must be rebuilt or restored from backup.

We’re not the only ones who have seen this behavior, either:

VMWare and NFS on NetApp Filers
NFS Datastores and what was their BIG issue…

The key takeaway from this is the following:

• If you haven’t yet applied ESX350-200808401-BG, you should apply it. Don’t wait. Now.
• If you did follow NetApp’s recommendations and set NFS.Lock.Disable, then review the instructions below (credit to Rick Scherer and several others, thank you!) to remedy the situation.

The instructions you should follow to restore NFS locking are here (these instructions include applying the patch):

1. Download patch ESX350-200808401-BG
2. Identify an ESX server against which you will apply the patch
3. Use VMotion and migrate the running VMs to other VMware ESX nodes
4. Install this patch on the selected VMware ESX server
5. In the Advanced Configuration settings ensure that NFS file locking is enabled with NFS.Lock.Disable=0 (you can also use the esxcfg-advcfg command to set this value)
6. Edit the /etc/vmware/config file and add this line:
   prefvmx.ConsolidateDeleteNFSLocks = "TRUE"
7. Save the changes to the file and reboot the VMware ESX host
8. Use VMotion to move VMs back to patched/NFS lock enabled/prefvmx enabled host
9. Repeat the steps above on each host in the cluster

So, again, if you haven’t applied ESX350-200808401-BG, please do so as quickly as possible, and then ensure that NFS locking is enabled. If you’ve previously disabled NFS locking, then follow the steps above to restore NFS locking and protect yourself against this possible split-brain scenario.

Related articles:

• VMware to release SMB virtualization kits by year’s end
• VMware tightens up virtual machine security
• VMware unveils 32MB version of ESX Server
• Virtual infrastructure: Threats lurk near open doors