Blogketing and Catch-22's

FYI - Since many hardware vendors are making a song and dance (and some of them little else) over the green issue I thought Id change the font colour for this post to green.  I imagine this is actually more than some vendors are actually doing ;-) 

Anyway – Barry Burke, the internets resident storage anarchist , has invented a new portmanteau – “blogketing”.  A contraction of the word blog and the word marketing.  Quite clever, and as you won’t find the word in any dictionary yet, Barry kindly provides a comprehensive range of real world examples in his own blog – just read any of his posts, he’s quite the expert.   But hey, who of us doesn’t dabble in a little blogketing from time to time ;-)

The strange thing is, it only seems like 5 minutes ago when some of us were bemoaning the lack of energy that HDS was putting into its marketing .  And here we are now, seeing them regularly lambasted for being overly energetic with some of their marketing.  Not to worry though, I imagine in due time they will master the art.  As for Hu, he may be a little unpolished when it comes to slipping in the marketing blurb.  However, I’m confident if he reads any of the EMC blogs out there, from the likes of Barry and Chuck , he will quickly learn the finer arts and hone that skill of subtlety :-D

Actually, did anybody else notice that Hu mentions the HP and Sun versions of the USP.  Im not sure Ive noticed him do that before.  Hmmmmmm!?!?

Anyway, enough of the banter, that’s not why Im putting pen to paper tonight….

Barry has recently nailed up a list of what he and EMC have learned about thin provisioning through a year or so of experience implementing Calerra.  He is referring to it as a list of catch-22’s for thin provisioning.  A list which he occasionally mentions and has asked for feedback on. 

A while back I was going to provide some feedback, but at the time I was too engrossed in series 1 of Heroes.  Now that I’m finished series 1, I’ve a whole load more time on my hands so here’s some feedback –

Barry states that bad things happen when somebody carelessly dumps their entire MP3 collection onto a thinly provisioned LUN.

Well, ….yes bad things certainly could happen….  I suppose it depends on your interpretation of bad.  Bad things will not happen to your critical systems unless you removed your brain while designing your environment and planned your pools and storage layout so badly that your mission critical systems share a common pool with your user’s home drives and the likes.  I don’t think you will find that design in any best practice guides (even for Calerra).  So the simple answer to that being; the use of multiple pools, good planning and good storage admins.  Nothing new there then.

Barry also states that bad things will start to happen if the PO for important additional storage gets held up, or the drives don’t arrive on time.

That one I have to agree with.  However, I have seen and worked in environments where delayed PO’s or shipments of disks have caused problems even with standard LUNs that aren’t making use of thin provisioning – if you need space and don’t get it in time you will be hurt.  So this is not a problem confined solely to thin provisioning.  Of course the situation could be made worse with the use of thin provisioning - everything sharing the effected pool could take the hit.  But remember, not all LUNs will be thinly provisioned and you will likely have multiple pools to mitigate this.  So again, good design and good management can help reduce the risk.

Along the same line, of bad things happening, Barry states that this will also affect “critical applications that can’t be allowed to EVER get a ‘disk full’ error”. 

Again, that would only be the case under an extremely bad design.  Of course, for the purpose of scare mongering, Barry will mention things like that in his blogvert, but in real life Im certain he would never suggest such a bad design.  I mean who in their right mind would thin provision LUNs to an app that “can’t be allowed to EVER get a disk full error”.  Even if you did, you would make sure it had its own pool so that it could realise the performance benefits of wide striping (Hitachi implementation of thin provisioning) while being protected from other systems misuse of pool space.

Barry then points out that deleting a file in most filesystems does not actually delete the file, never mind allow the space to be reclaimed by arrays supporting thin provisioning.  He also points out that filesystems often avoid re-using space occupied by deleted files until there are no other free blocks  available.  Essentially, from the OS point of view it may appear that less pool space is being used than actually is.

I agree again, in part.  I just think this kind of thing can be easily overcome with the use of an agent on the host that provides a mor e informed view to the server administrator – include them in the wider picture….. you know, talking to the other guys in the IT department and working like a team ;-)

I also see that HP are talking about the EVA supporting new functionality in Windows Server 2007 that enables shrinking of LUNs……  although I haven’t had a chance to look into this, I wonder if it could provide a solution to the issue of reclaiming unused space in the future.  May be somebody more informed can elaborate on this Windows Server 2007 functionality?

I also see a plus side to not reclaiming space occupied by deleted files, but Im saving this one for later.

Along the a similar line he mentions that Windows Vista keeps copies of multiple versions of files on disk to assist in restores, consuming more pool space.  

Firstly I don’t know of anybody SAN attaching Windows Vista.  But when they do, either disable this feature, or if that’s not possible, don’t thin provision LUNs to those hosts until it is possible.

So far not much that can’t be avoided with good planning and execution……

Barry then says that performance can be significantly reduced by thin provisioning LUNs

Well….. show me a storage admin task that doesn’t affect performance when executed badly?  Again nothing new there.  On the flip side however - if you do thin provisioning right, at least on Hitachi storage, performance can actually be improved! 

Barry bases his point on the premise that improved storage utilisation boils down to storing more data on the same number of spindles.  Obviously disks have a limited number of I/Os they can service, and squeezing every last inch of space out of a disk will often push a disk past its I/O limits.

He actually gives an example to illustrate the point.  But what he doesn’t seem to have noticed that he has already provided us with a solution ………  Earlier in his post he made several references to unwanted data being dumped onto thinly provisioned LUNs - user dumping MP3 collection, routinely deleted files, may be even runaway apps that don’t need to keep all of the data.  He has also pointed out that this space cannot easily be reclaimed by the array.  Now although this may at first look like a bad thing for thin provisioning, it can also provide some performance benefits. 

Basically many hosts will consume an amount of space that they don’t really use.  Beautiful in its simplicity (err well…).  

So we have a problem providing a solution, does that qualify as circular logic and make it onto the Catch-22 list?

And if you create your thin provisioning pools from slices of physical disks and use the other slices for normal LUNs, you won’t hit the disks inherent IOPs limit so quickly, if ever.

And what about the trend of larger and larger disk drives?  They bring with them the very same IOPs problem that Barry is talking about – as drives get bigger we tend to put more LUNs on them and demand more IOPs from them, but their underlying I/O limitations do not increase in concert with their size.  I haven’t seen Barry bemoan the fact that the Symm supports huge drives which bring the same problem.  I seem to be the only one who cares about that !  Go figure.

Anyway, time to wrap this post up.  In my opinion, Barry has painted a picture of thin provisioning, and at first glance it might look good, may be even a masterpiece.  But to the discerning eye, there is a lot of white space left untouched.  May be he knew that he hadn’t finished the painting, after all he did ask for feedback and was no doubt trying to add some balance to the debate at the time.

I do wonder if this will end up being like the RAID6 debate?  “RAID6” being pretty much a cuss word to many an EMC’er.  Until they joined the party that is

And although my knowledge of the thin provisioning is based on the Hitachi implementation, lets not forget that there are far more established thin provisioning players out there and they seem to be making a success of it.  If you read Anil Gupta’s blog you will see that he is seeing that 3PAR has a lot of satisfied customers.  So thin provisioning has to work for some people.

Nigel