FC: recapping FC flow control

Amid all the discussion of FCoE and the many requests to cover more aspects and underlying technologies, I thought it probably a good idea to take a step back for a moment and recap flow control in native FC networks.  After all, FCoE is almost a transplant of FCP onto Ethernet.

Granted, to some people this will be teling you how to suck eggs, but for others clearly not!  So………. native FC Class 3 networks achieve reliable frame delivery/lossless behaviour through the use of a link layer flow control mechanism called buffer-to-buffer credits (aka BB_Credits) 

NOTE:  Class 3, in FC parlance, basically refers to a connectionless network without acknowledgements.
Class 3 is the most common class of service in todays FC SANs.  It is connectionless without notification – somewhat similar to UDP in LAN environments.  However, and very importantly, if a SAN is well designed and quality components and installation practices used, the lack of acknowledgements is not an issue – in fact it’s actually a bonus.  Why have an ACK related overhead when the network does not lose frames.

At a high level, the FC BB_Credit scheme works on the principle that a sender cannot send frames unless it explicitly knows that the receiver is able to receive and process it.  This behaviour ensures that congestion does not arise and therefore frames will not be discarded.  See the diagram below -


NOTE:  I should point out at this stage that link layer flow control in Enhanced Ethernet networks is based on PFC (802.3Qbb) and not BB_Credits – see my previous post here.  This is one of a small number of differences between FCoE and FC, and as a result BB_Credits are not part of the FLOGI process in FCoE networks.

The FC BB_Credit system is a hop-by-hop system, it has no notion of an end-to-end connection and only works between N_Ports and F_Ports and not between two N_Ports.

Each device on a native FC network establishes its initial BB_Credit pool with the fabric switch that it logs in to (FLOGI).  E.g. An HBA that is plugged in to the fabric and establishes an initial BB_Credit pool with the switch port it logs in to.  Each BB_Credit then represents a single frame that the HBA is able to send.  Correspondingly, each time a frame is sent, the sender must decrease the number of BB_Credits in its pool by 1. 

The number of BB_Credits a device can issue to connected devices is based on how many buffers they have (HBAs and switch ports have physical buffers, send buffers and receive buffers).  More buffers = more credits.  However, more buffers also = more ££$.

At the other end, receivers transmit R_RDY primitive signals to notify senders of the availability of receive buffers.  This is the means by which senders replenish their BB_Credits.

If senders only had 1 buffer credit and had to wait for that to be replenished after every transmit, this would lead to horrendously low link utilisation.  To more efficiently move data on the FC SAN, multiple frames are sent consecutively so that senders do not have to wait for their pool to be replenished after every frame sent. 

Data can be sent on the transmit line and credits replenished, via R_RDY primitives, on the receive line in full duplex fashion.

And FCoE?

Although the FC Class 3 buffer credit system is different to Priority based Flow Control (PFC) in Enhanced Ethernet and used by FCoE, the end result is essentially the same – a lossless network!  In fact, if the truth be known, the FCoE system of using PFC is actually superior

It appears that people have a natural tendency, especially FC folks, to associate Ethernet with congestions and routinely discarded packets.  But such comments and feelings are referring to your grandmothers Ethernet, and we need to come to understand that FCoE networks are not built on your grandmothers Ethernet, they are built on Data Centre Ethernet/Converged Enhanced Ethernet – which is another beast altogether.  In fact I may go so far as to suggest that if they have Ethernet in heaven it will be DCE/CEE!


You can follow me on Twitter (I only talk about storage and virtualisation) –
http://www.twitter.com/nigelpoulton – or @nigelpoulton 

You can also reach me at nigel at rupturedmonkey dot com – Im available to hire as a freelance consultant