SATA in a USP V. As rare as hens teeth

This post is for anyone interested in Hitachi storage who feels like they’ve had enough HAM for a lifetime and just wants a taste of something different for a change….

One of the projects I’m currently working on involves deploying a new USP V with native SATA disks side-by-side with FC disks (when I say “native”, I mean the SATA is sitting inside the USP V and on an external array being virtualised with UVM).  Although this configuration has been supported for a while, it’s the first time I’ve seen or done it, and hence the first time I’ve really given it any thought.  So I thought I’d share my thoughts for the greater good.

Seeing as how HDS and HP have supported SATA in a USP V/XP24000 for a while now, you’d think there’d be a bunch of people out there with experience and a boat load of best practice advice doing the rounds, right?  Apparently not!

In fact, the only best practice advice I initially got was that RAID 6 was recommended for the SATA disks.  Fair enough, in a storage array that builds RAID Groups in the way that the USP V does, RAID 6 is a no-brainer for large SATA.  But even knowing what little I know about SATA and the USP V I couldn’t believe that there was no other best practice advice.


Concerns

My overall question was simple –

What is the impact of installing larger, slower disks running a different protocol and a potentially higher failure rate, side-by-side with smaller faster FC disks?

When I asked this question, the most I got back was “you need to do RAID 6 with SATA”.  Thanks, but to be honest not very helpful.  Certainly not good enough for me to press ahead with a production install. 

So to break down my concerns into more detail –

1.  What is the recommendation regarding cache partitioning?

2.  Is the imposed verify-after-write operation that is mandatory for all writes to SATA disk in HDS midrange kit (AMS) also a requirement on a USP V with native SATA?

3.  What is the potential impact of larger, slower and disks with a potentially higher failure rate and resultant elongated rebuild times?

4.  Is there a recommendation for backend layout – SATA sharing BEDs with FC or SSD?

5.  Are there any System Option Modes (SOM) recommended for SATA disk, and if so what are they?


Really simple, and to me, intuitive questions.  Boy was it hard to get people to take me seriously!

So as a result of some digging, I propose the following answers and loose best practice advice –

NOTE:  The following are my opinions and I am absolutely not an authority on the USP V or XP24000.


Answering those concerns

Q1.  Cache partitioning

I highly recommend a dedicated cache partition for SATA.  This is to provide an element of protection for global cache and other cache partitions, effectively ring fencing a portion of cache for SATA. 

Why?  The potential for high cache write pending (WP) is increased with the use of SATA due to the larger size of SATA disks, the slower rotational speed, overall lower throughput, as well as the more limited protocol.

Obviously the size of you cache partition (CLPR) is entirely dependant on the amount of capacity in the SATA pool as well as how hard you will be pushing the SATA.  However, it is important not to size the CLPR too small as doing so can increase the chances of high Write Pending in the SATA CLPR, which in turn can affect the rest of cache (more on this in my answer to question 5). 

A good approach might be to start with a relatively small CLPR (but not too small) and grow it if required.  You can grow CLPRs in increments of 4GB relatively non-disruptively and easier than trying to reduce a CLPR.


Q2.  SATA and Verify-After-Write

In the HDS midrange kit, such as the AMS, there is an imposed verify operation (read) after every write to SATA.  This operation is intended to ensure your data is correctly committed to and retrievable from disk.  However, this brings with it a large performance penalty – increased number of backend read operations for every write operation.  This performance penalty saw SATA in HDS midrange kit perform quite poorly so far as feeds and speeds are concerned, but obviously perform admirably when it comes to data integrity.  Not the end of the world when we consider that SATA is not intended for high performance applications.

The USP V on the other hand is designed to be a high performance array.  So implementing a verity-after-write operation on a USP V in the same way as it is implemented in an AMS array would be disastrous for performance.

Fortunately it just so happens that the guys at Hitachi knew this, and as a result decided to implement the verify operation in circuitry in the disk canister.  This has the positive effect of keeping the verify operation without imposing extra load on the back end directors (BEDs) and back end loops.

It is probably worth pointing out at this point that SATA disks supported in the USP V have an FC <-> SATA bridge in the disk canister allowing them to sit directly onto the existing backend infrastructure, side-by-side with FC.

Net result is that in many cases SATA and FC can realistically site side-by-side on the same BEDs and loops.

But before jumping in with both feet, as always, consideration should be taken with performance centric configurations.  For example, a high performance OLTP System requiring its own dedicated 146GB 15K FC or 146GB SSD (assuming high random read) is probably not best suited sharing its backend with SATA.


Q3.  Potential impact of larger slower disks ……

While this is quite a generic question, the biggest risk, in my opinion, is around the elongated rebuild times associated with large SATA disks.  Rebuilds cause I/O on the backend loops as well as overhead (CPU cycles etc) on the BEDs.  The longer a rebuild, the longer this increased load exists.  On a subsystem under load, rebuilding large SATA disks can take several days.  I don’t know about you, but I don’t want my high performance LUNs sharing a backend with this.

Also, I know that SATA is usually associated with higher failure rates than FC.  However, I have to be honest and say that this is not something I have actually noticed in the real world.  Let’s not forget that build quality for SATA disks, bearings and all, has improved and is of a higher standard for enterprise class SATA.  Couple this with the fact that the disks will be housed in an enterprise array that is designed from the ground up to be a disk friendly environment – think; power, cooling, humidity, vibration……  MY personal opinion is that the notion that enterprise SATA disks fail frequently is a bit of an urban myth.  Happy to be proved wrong…. Well not “happy”.

Elongated rebuilds also increase the window of exposure to a double disk failure.  However, RAID 6 effectively mitigates this.


Q4.  Sharing your backend with SATA

This is issue is somewhat put to bed with the implementation of the verify operation in the disk canister.  With the responsibility for the verify operation being removed from the BED to the disk itself, backend performance should not be hugely impacted by SATA.  Therefore, is it moderately safe to have multiple Tiers of disk share the same backend.  However, having said that, as previously mentioned - I would not personally recommend putting your highest performing disks on the same backend as SATA.

However, whether or not you want SATA stealing your premium priced USP V resources such as cache slots, CPU cycles, internal bandwidth and backend IOPs is another question.


Q5.  Are there any recommended SOMs for SATA

As appears to be standard practice in most installations, System Option Mode 454 should normally be set to ON.

This will base destaging algorithms etc on the average WP rate of all cache partitions, rather than just the highest.  Let me give an example –

You have two cache partitions; CLPR-0 and CLPR-1.  If SOM 454 is set to OFF and either of the cache partitions reaches 70% write pending (WP), aggressive destaging will kick-in across all partitions.  However, if SOM 454 is set to ON, destaging algorithms will be based on the average WP rate across all cache partitions.

Other than this, I’m not aware of any other SOMs that are of use particularly for SATA.



Resultant best practices

So my own personal best practice for USP Vs deployed with internal SATA are as follows –

• RAID 6
• Dedicated CLPR for SATA
• SOM 54 turned ON
• Isolation of high performance Array Groups on BEDs with no SATA disks.
• Tier 2 FC Array Groups can probably co-exist side-by-side on the same BEDs without any conflicts.