FCoE: Framing deep dive
My current series of posts on FCoE and CEE have generated a lot of questions and feedback via the comments section and on Twitter. In this post I will attempt to address some of the questions and misunderstandings – especially around encapsulation and framing efficiency……
I received a comment on the site today from a confused reader. The point of confusion is a common one and not something to be embarrassed about. Part of the comment read as follows
“…..but it’s [FCoE] problem is that it consists of 3 different protocols (IP, FC, SCSI) just enlarging it’s frame size with unneccessery information. This is a step backwards."
The above is clearly a misunderstanding. It would be accurate to say that DCB/CEE is designed to transport all three of the above protocols, but FCoE does not consist of these. So if it doesn’t consist of these then what is it…….
FCoE in a single sentence
Put in to a single sentence “FCoE is the encapsulation of Fibre Channel frames within Ethernet frames”.
.jpg)
If you have a good quality dive suit then put it on…… we are about to perform a deep dive…………..
Squeezing it in
Some will know and some will not, but a standards based Ethernet frame with VLAN Tagging, including Tag Control Information can be 1522 bytes in length. On the other hand a Fibre Channel frame can be up to 2148 bytes in length. Clearly, this poses a problem – How can an Ethernet frame be wrapped around a fibre channel frame, if the fibre channel frame is bigger?
Two of the most viable and obvious solutions to this challenge include –
- Fragment the FC frames so that they fit within standard Ethernet frames.
- Make bigger Ethernet frames
From a performance and simplicity point of view, option 1 should be avoided at all costs due to the fact that fragmentation adds complexity to the encapsulation and extraction process. Complexity brings overheads, which in turn would slow the protocol down - which if we remember back to my initial FCoE post, is highly undesirable in networks transporting SCSI as a payload.
Option 2, on the other hand, keeps the encapsulation/extraction process very simple – 1 FCoE frame for every FC frame.
No fragmentation and reassembly overhead, no tinkering with the headers or payload, and vitally, no unnecessary protocol induced latency. These are big wins for networks transporting SCSI. This seamless handling of frames offered by option 2 keeps the encapsulation/extraction process very simple, very lightweight and ultimately, very fast!
To put this in perspective, here is as good a place as any to point out that the encapsulation and extraction process occurs at every hop on the FCoE network (we won’t talk about multi-hop FCoE here). In other words, every time an FCoE frame traverses and FCoE switch, it is re-encapsulated ready to be moved on to its next hop.
NOTE: Due the broadcast nature of the Ethernet networks, frames need a new source and destination MAC address for each hop. Point being, if the encapsulation/extraction process was arduous, it would significantly impact the performance of FCoE.
Option 2 also keeps hardware designs simpler, allowing for cheaper hardware than would be required if complicated encapsulation and extraction techniques had to be implemented.
Fortunately the folks on the standards body (INCITS T11 FC-BB-5) decided on option 2.
Jumbo to the Rescue
But in order to go with option 2, we need to invent larger Ethernet frames. Well actually we don’t need to invent them because they already exist and are called Jumbo Frames. Baby Jumbo frames (~2.5KB) provide enough space, plus some headroom for FCoE frames.
The diagram below shows an FC frame encapsulated within an FCoE frame.
Please feel free to correct the above diagram if its incorrect or unclear….
Although Jumbo Frames are not an officially recognised IEEE 802 standard, in fact due to technical reason they likely never will be, they are extremely widely implemented, well understood and about as standard as anything non-standard can be. In fact, if jumbo frames did not exist, they would have to be invented to enable FCoE networks.
The same comment that came in from a reader talked about poor framing efficiency in FCoE. Lets take a quick look at this while we’re here.
Data is protocol overhead
By “framing efficiency” we mean the number of control bytes sent versus total number of data byes sent. A protocol with a high percentage of control bytes would be seen as inefficient. Because of going with option 2 from above, FCoE actually has quite good framing efficiency, as it doesn’t carry baggage required to reassemble an FC frame from multiple FCoE frames.
I’ve heard it said that to techie people data is protocol overhead ;-)
Keep your hands off my bandwidth
Another reader has recently asked the following question
“….how is the IP traffic and the FCoE traffic kept from consuming one another’s bandwidth…”.
The answer is to do with Ethernet Priorities and I touched on them in a previous post and in the comments to that post, but while we’re here with the diagram above lets add a bit more meat to the answer.
I’ve written VLAN Tag in the above diagram, lets break it open a little more –
The above is a representation of an 802.1Q VLAN field. This field also lists the Ethernet Priority (technically referred to as the Priority Code Point or PCP). This is leveraged by Priority based Flow Control as discussed in earlier posts and is absolutely critical in the implementation of a lossless fabric, without which FCoE would be dead in the water. PCP can be between 0-7 and each type of traffic will be encoded with its own PCP allowing features such as losslessness and bandwidth allocation etc to be implemented. FCoE would have one value, iSCSI another etc…..
Hope that helps clear some things up. Let me know if you have any more questions. I’m enjoying the discussion that this is generating, and don’t be embarrassed about asking questions(obviously I don’t know all the answers but Im sure somebody out there will).
Nigel
I am currently on the market for FCoE related work and can be contacted at nigel AT rupturedmonkey DOT com