As interesting as funny puppets are, and if someone actually took the time to read my Blog entries as a cumulative collection of posts and not just what is on the front page, they will see I’m usually un-biased.  In fact I specfiically call out NetApp as one of the lead NAS vendors in several areas, including having the SPEC-SFS results that showed the OnTap GX cluster as by far the IOPs leader for clustered file systems.  That artcile, by the way, is far from “second rate blog” material, thanks for making it personal, not all of us are paid to pick on others products, some of us actually have the experience in technological reality, developing, delivering, and seeing products within customer accounts.

     Now let’s take a look at Alex’s blog response.  It is funny that I took a statement about NetApp picking on 3Par’s solution, and it turned into a “They must be very worried indeed”, the THEY in this being Dell of course.  This was never about PS-series storage or Dell for that matter.  So, yes of course I do what anyone with literary license does, I took a graph from a public NetApp document and posted it showing clearly that the WAFL has an issue when utilization is increased in a volume. (hey Alex, yes this was originally about utilization claims in a performance benchmark).  So from that same document Alex pulled the following:

     With the “claim” that concurrence fixed the problem.  Well what isn’t shown, and was neatly left off this graph was what the “performance” looked like “with concurrence” at utilizations prior to 50% (notice the green line for those following).   So, if NetApp would like to publish what this graph looked like before it was edited (or maybe those tests were never run)…

     Leaving me to defend Dell’s, the leading iSCSI storage vendor (by market share according to Gartner), PS-series storage arrays and documented benchmarking…….which I am un-willing to do, as this isn’t about Dell, or Equallogic.  This was my personal defense of Marc Farley, who I’ve known for awhile and felt his post was being picked on.

So lastly, as I’m kind of on a graph roll here, Alex posted the following:

     Which was so blury you can’t even make out what is bei’ng represented.  However, what this is, is showing that “over-time” IO has remanded constant in a “high utilization” environment with Snapshots enabled.  So what is missing here, is, not a change in utilization, because we’ve already proven that with NetApp you loose performance as you increase utilization.  It also shows that NetApp has LITTLE TO NO OEVERHEAD with Snapshots!!!!!!

     Hold on your your hats here….YES IT IS TRUE NETAPP HAS SNAPSHOT TECHNOLOGY THAT IS TRUELY LOW/NO OVERHEAD!   I said it, the pain from the Dell implant almost kept me from typing those very words.  Why is this true?  Because, NetApp DOES NOT USE a COPY-ON-WRITE based snapshot technology.  This is a very good thing, since copy-on-write based snapshots incur a pretty hefty performance penalty.

I came across this posting by NetApp’s Alex McDonald trying to pick on up-start 3Par’s recent SPC-1 benchmark release and Marc Farley’s blogging about it.  The reality is NetApp is very concerned about how people state performance.  Why?  It is because in NetApp’s OWN Technical Reports they show over a 50% reduction in performance as a Volume/LUN data utilization passes greater than 60%.  You heard that right, they show support for up to roughly 25000 users at 10% capacity of a Volume, and only roughly 12,500 users as that same Volumes capacity reaches 60%.

Say it ain’t so….(cries OnTap users everywhere and some old baseball fans)

As extracted from NetApp’s Technical Report comparing performance against EMC storage:

So what is WRONG with Alex’s thinking?  Alex makes the comment:

Anyone see the problem here? 83% capacity utilisation? And with all the disks mirrored?

200% Surprised

How do you get 83% utilization when everything is mirrored? Heck, how do you get anywhere near 50%?

Of course Alex wants you to think that 3Par is only “utilizing” less than 50% of the storage.  However, regardless of disk architecture behind the scenes, they were in fact running the VOLUME at 80+ percent capacity, which is exactly the public claim the announced.  They didn’t claim 83% of the storage system was utilized, or 83% of the RAW storage allocated, it was capacity utilization of the LUN being benchmarked.  Alex was looking for a comparison against a RAW storage number, not usable storage, however, a RAW storage number has no place in this conversation.  Also, believe me, having the data mirrored does nothing for increasing performance of any solution.

If you take written data(utilization) as a percentage of RAW capacity for ANY storage vendor’s benchmarking then there will be some very interesting results out there.  Even the above graph from NetApp themselves would look MUCH worse then it already does.

“So Long, and Thanks for All the Fish!”

I think the best vendor marketing campaign in recent storage history has been the full force attack of NetApp against iSCSI for use with VMWare.  Now, I can fully understand NetApp not wanting any customers using their “iSCSI”,but iSCSI in general isn’t slower than NFS for VMWare.  How do I know this?  Well common sense for one, but let’s not bring my own personal emotions and experience into this conversation.

I think the best part of this marketing campaign was the release of a “white paper” or “tech report” or “case study”, whatever you want to call it.  It was NetApp Technical Report TR-3697 – Performance Report: Multiprotocol Performance Test of VMWare ESX 3.5 on NetApp Storage Systems.  I think the title explains the problem with this idea from the very beginning.  The purpose of this report was the following:

Both NetApp storage arrays and VMware® ESX 3.5 natively support data access using three storage protocols: FC, iSCSI, and NFS. Due to the protocol flexibility and neutrality of both companies, many discussions have been had around the water cooler comparing the administrative pros and cons of the storage protocols as well as how they compare from a performance perspective in an ESX environment. This technical report, completed jointly by VMware and NetApp, shares the results of testing conducted to compare the performance of FC, software-initiated iSCSI, and NFS in an ESX 3.5 environment using NetApp storage. The results compare the performance of the three protocols with a goal of aiding customer decisions as they build out their virtual infrastructures.

Clear as day right?  Well what is my problem with this approach?  While NetApp does in fact support FC, iSCSI and NFS, they cannot claim to be the fastest FC, nor iSCSI block storage devices, and most likely are not the fastest NFS device on the market, however they are a NFS/CIFS device first and foremost.  Why do I point this out?  I’ll refer to an NetApp TechOnTap article.  Within this article it points out that:

Block Storage Protocols
Initial NetApp storage systems were NFS appliances. Over the years, the same core
architecture has been extended to support multiple protocols—CIFS initially, and then
block-based protocols (Fibre Channel and iSCSI). Block protocols expose LUNs, which
are special WAFL containers (“files”) that exhibit block device characteristics. They
inherit the rich lineage of WAFL—including space- and resource-efficient Snapshot
copies and clones.

So what does this mean for an end-user?  It means that when they create a FC or iSCSI volume on a NetApp filer, they are creating a very large file or “special WAFL container” that is virtualized out of the filer head as a raw block level device.  Which, of course, an OS then needs to mount through another volume manager and then place a whole other file system on that.  Clearly several layers of translation will increase latency and IO throughput.

So, where am I going with all of this?  Clearly, NFS is going to be one of NetApp’s best performing protocols, because they have to build everything else on top of the file level architecture.  Which leads me to my final comments.  Released in a session at VMWorld 2008 in Europe was a direct comparison of NFS vs. iSCSI as far as performance.  The title of the presentation is:

NFS & iSCSI: Performance Characterization and Best Practices in ESX 3.5

In this document iSCSI storage beat out NFS storage in performance, and with lower latencies.  Take a look and see for yourself.  So, when choosing a storage vendor, do some research, don’t just believe a “Tech Report”.

Comments always welcome….

Al Franken, thank you for giving me the title for this blog entry. For those of you not familiar, Al Franken was a major comedic contributor to SNL (Saturday Night Live for those of you living in the dark ages) and that quote was a title of one of his more recent non-slanted liberal musings. This brings me to my topic of the day, Benchmarks and Benchmarking.

Coming front the world of clustered file systems, specSFS seems to be the standard benchmark that is easily available to access and is the mostly widely abused benchmark of all times. Now before the spec guys get upset with me, the problem isn’t with the specSFS tool-set, the problem is with the vendors who use it (I am guilty of this vendor deception personally).

For those of you not familiar with this benchmark, it is specifically used to test NAS IOPs performance. Some of the best performance ever seen by this benchmark was most recently released by Network Appliance, Inc. They were able to invent a solution that pushed the specSFS benchmark to the next level of performance, the 1,000,000 IOPs solution. Why this is irrelevant I will explain as simply as $. Actually, closer to $,$$$,$$$, please input any 7-8 figure dollar amount you wish, because that is the cost of 1,000,000 IOPs utilizing ONTap GX clustering configurations.

I am going to breakdown the top 5 producing solutions in this entry. This will be broken down into the following categories:

1. The solution being provided.
2. The Posted Result
   1. IOPs
   2. Latency
3. Load generation
   1. Number of Servers Required
   2. IOPs per load server
4. File System(s)
   1. Number of file systems (anything more then 1 is bad)
   2. IOPs per file system
5. Disk
   1. Number of Disk Controllers
   2. Number of Spindles
   3. Speed of Spindles
   4. IOPs per controller
   5. IOPs per Spindle
6. Relative Cost of Solution
   1. List pricing for components
   2. IOPs per $

Before I get going, I’m going to try my best to come up with list pricing costs for these solutions. I haven’t ever ordered these products and everyone of these vendors seem to have pretty complex configuration quoting tools. I’ll do my best!

THE LIST

(This list was based on specSFS results posted prior to 9/01/2007)

1. Network Appliance, Inc. – Data ONTAP GX System (24-node FAS6070)
2. EMC Corp. – Celerra NSX Cluster 8 X-Blade 60 (1 stdby) 2 DMX
3. Panasas, Inc. – ActiveScale storage cluster (60 DirectorBlades)
4. Exanet, Inc. – ExaStore EX600FC
5. BlueArc Corporation – BlueArc Titan 2200, 2-Node Active/Active Cluster

For those of you who aren’t interested in a LONG read, here is the summary in table format.

  Read the rest of this entry »