DCE and FCoE: Of Roads and Rail
I see Snig has also posted on this topic earlier today and made some interesting points and there may be some overlap.... no harm though.
I started writing this post a few weeks ago as a knee-jerk to Ciscos Unified Computing announcement, but scrapped because it got too long. Todays rehash of the same thoughts is hopefully shorter and of interest.
For a while I’ve had a bit of a knot in my gut over the whole Data Centre Ethernet and Fibre Channel over Ethernet thing(s) – I can’t decide if I’m excited or about to soil my kegs – a bit like at the start of a roller coaster ride. To be honest I think a bit of both, although I think it’s healthy to have a twinge of fear about something new to keep yourself in check – I saw a programme on TV not long ago about a guy who was killed by his own Boa Constrictor snake, apparently he got too comfortable around it and dropped his guard one day, sadly it killed him. So there is definitely something to be said for having a healthy respect for something. Any similarities between a Cat5 cable, commonly used by Ethernet, and a Boa Constrictor are purely co-incidental ;-)
The beast is at large
One of my main concerns with the trend towards convergence to Ethernet is that I see it as another tentacle belonging to the beast that is prowling the streets of the IT world squeezing more and more out of less and less – headcount culls despite workload increases, bigger disks without greater IOPs power....... We see it all around us in IT.
Of course such cut backs and squeezing are at times necessary and have their benefits, however, we need to be sure we don’t go too far.
Do we expect a single cable and a single CNA super NIC (Converged Network Adapter) to be able to replace multiple NICs (prod, backup, vmotion....) as well as fibre channel HBAs? That’s a lot of eggs in one basket, and can the internal architecture of today’s or even tomorrows servers drive these CNAs? Well may be it won’t quite work like that but lets be careful we don’t expect too much.
Recently Steve Duplessie posted about Cow Economics and related it to storage (its worth a read). If we were to apply the Cow Economics model to convergence to Ethernet we might get something like the following –
You have two cows (Ethernet and FC infrastructures)
You stop loving one (FC) and leave it rot in a far away field
You make the other do the work of both.
At first, all is well. But after a while it becomes weary, develops mastitis and all manner of nasty infections due to oeverwork and no rest. You cant go near it anymore without risking a hoof in the chest. You start to hope the other cow that you left to rot in the far away field is still alive.
FCP and IP – worlds apart
FC (both FCP and associated FC infrastructures) and TCP/IP were created to address vastly different needs. TCP/IP was designed to be highly resilient and to work on unreliable mediums, and as a result doesn’t make too much of a fuss over dropped packets and the likes. FCP on the other hand does not like it when her needs are not satisfied and will introduce you to a world of hurt if you dare to drop her packets.
Now this seems like a goo place to point out that I know that FCoE supposedly has very little, some would say “nothing”, to do with IP and everything to do with Ethernet. However, in my opinion it is not realistic to think we will be able to throw them both on to a shared medium and not have them tread on each others’ toes once in a while. Let me give an analogy that I think illustrates my point -
The Open Road
Not that many years ago the British Motorway network was envisioned and built. At the time, the government ran adverts with titles such as “Welcome to the open road” emblazoned atop pictures of brand new roads with no other traffic in sight.
Lovely. But that was back then. Anyone who uses the motorway network in this country knows only too well that this is a far cry from the carnage that prevails on most motorways in the UK today.
But what has this to do with convergence to Ethernet and FCoE? Well, like the convergence to Ethernet, the motorway network handles all kinds of traffic, from lumbering articulated lorries, through; motorbikes, caravans, wide-loads, and even my mum chugging along at dangerously slow speeds.
And no doubt it all worked perfectly back in the day, but as more and more and more and more and more traffic has been introduced to the roads they have started to creek. Sure there is prioritisation - the lorries can’t use the outside lane, are restricted to 56mph and the likes......... but the facts are , you throw enough traffic (no pun intended) at a shared medium, and sooner or later you have contention, and mayhem. And if there’s an accident, even if it’s in the other direction, or if they need to dig a lane up, you can kiss goodbye to the next hour or so of your life.
Reaping what you sow and getting what you pay for
Obviously I’m not suggesting that we should rip out the shared motorway network and go back to the old days. But I am saying that if I want to commute to London or Glasgow I will take the train, on its dedicated infrastructure, despite the price premium. When I take the train I escape the nightmares of the shared motorway network - I generally sleep on the early outward journey and watch a film on my laptop on the way back - bliss. A world and a half away from the concentration and battling required to successfully traverse the motorway network.... As a general rule you get what you pay for (thats my long winded way of saying “yes FC is an additional cost but there is a reason for it, and often it’s worth it”).
Now I know people are talking about cost savings from DEC and FCoE, but any real cost savings are surely a long way down the line. Implementing new infrastructure, technologies and getting people with the appropriate skills will surely add a cost premium up front. The latest and greatest hardware and associated management skills are rarely cheap. In fact, the only immediate benefit I can see, other than for Cisco, is a financial benefit for the people who will be hired to plan, implement and support early deployments. <Excuse me for a moment while I go blow the dust off my Cisco networking credentials .....> 
Simple = Reliable
One of the beauties of FC is its simplicity. It’s a flat network with very little overhead and relatively few complicated configuration options. Plug your targets and initiators in and configure the zoning, if you pass Go collect £200. It really can be as simple as that. Maintenance is fairly simple due to multi-pathing and the fact that FC networks are comparatively small tidy and controlled. More often than not, Simplicity = reliability. The corollary being complexity = unreliability, which can lead to the dreaded “downtime”.
Adding FC to the LAN will increase complexity, of that there can be little doubt.
As an example - anybody who works in storage knows of the complex support matrices that all vendors have - only certain HBAs with certain firmware and certain drivers are rubber stamped to work with certain switches or storage arrays, each again with their own firmware etc. Such complicated support matrices often lead to moderately frequent code upgrades on kit, including switches. Again, anybody with experience in storage knows that code upgrades that go bad on switches (be it occasional buggy code or unprepared engineers etc) can cause mayhem and sometimes downtime. Throw all of this in to the mixer with LAN and WAN, which are bigger more complex and adhoc than small and neat FC networks, and life is about to become a whole lot more painful when it comes to certain changes and upgrades (not just switches, but possibly firmware and driver updated for CNAs).
Oh and of course implementing new tech is nearly always disruptive, and DCE and FCoE are new tech.
Second class citizen
Is IP, or more accurately its customers, willing to roll over and become a second class citizens in its own house? FCP demands a lossless network and will requires lower latencies than IP. FCP will effectively be walking into IPs home, dirty boots and all, taking the TV remote away from IP, plonking itself on the couch putting its dirty feet on the linen and flicking to its own favourite station.
Aren’t you jealous of the people with prepaid electronic passes who whiz through the toll booths on toll roads while you queue with your coins. That will be IP watching the FCP frames whiz through the tool booths.
Also, storage demands, especially for lower tier storage, are growing at an alarming rate. Every server and his dog is now in the game for SAN storage. Thinking back to the motorway analogy – back in the day only the more well-off could afford cars and as a result there weren’t that many on the road, now every family has a car and many have more than one. Getting back to servers and storage, not that long ago it was a selective few servers that qualified for SAN storage, now every server being built requires SAN storage. Servers getting SAN storage is going in the same direction as car ownership has – up! And fast!
Tyre marks on the lawn
At this point it’s probably worth mentioning that we’re going to need IBM and HP to step up to the mark and ensure that Cisco don’t gain a monopoly share on Data Centre Ethernet. I actually have nothing against Cisco. What I do have a problem with is any single company having a monopoly on a market segment (think IBM and Mainframe). Competition is what makes the world go around.
I think there is weight to the notion, in part at least, that Cisco recently parked its car on HPs lawn in response to HP already having parked their car on Cisco’s lawn. However, from where i stand it looks like Cisco’s server platform will be competitive, can the same be said about HPs network platform? Is HPs car anywhere near as big as Cisco’s?
I will also be interested to look further into Brocades offering in this space although I think that is one hell of an uphill battle for them, but good on them for keeping things competitive.
Conclusion
If Cisco is the Borg then DCE and FCoE is Skynet, no matter what you you cannot stop it being born.
So no matter what we do, DCE and FCoE are probably going to change things in a big way sooner or later. However, we can do a lot to make sure it happens smoothly and in the right way. A general rule in life being “learn from mistakes, and if possible, others mistakes”. May be we can learn from the motorway and rail networks of Britain –
- If only they had built the service stations further away from the road and put the feet supporting the bridges and overpasses further apart. This would have made life so much simpler and cheaper when adding more lanes. Let’s make sure we design and deploy DCE Ethernet in a way that makes long term use and expansion simpler.
- While the motorway network almost killed the rail network, we are all grateful it didn’t. The motorway meets most of our travel needs, but sometimes the train is by far the better option. May be it’s a good idea for us not to banish our FC infrastructure to a remote field out of sight and leave it there to rot. One day in the future we might need to make use of it again.
With good planning and an ounce or two of strategic thinking (not 100% £$ motivated), we can make this work and it can be a fun journey too. Although on the surface this may seem like Im negative about DCE and FCoE, but I’m not – forewarned is forearmed. I plan to update my skillset accordingly and be ready to take the bull by the horns. Bring it on!
Nigel